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SUMMARY 


M 

This  research  develops  a multistage  decision  process  designed  to 
obtain  the  maximum  amount  of  information  from  the  evaluation  of  a factorial 
design  while  minimizing  the  amount  of  resources  used  in  obtaining  the 
information.  The  use  of  screening  experiments  in  building  the  factorial 
design  is  investigated  in  order  to  maximize  the  amount  of  information 
gained.  The  use  of  sequential  analysis  procedures  to  terminate  experi- 
mentation at  the  earliest  possible  time  is  investigated  in  order  to  mini- 
mize the  amount  of  resources  used.  The  research  is  limited  to  factorial 

2 r.'  ' ■ < 

designs  involving  univariate  response  models  assumed  to  come  from  a normal 
population;  however,  the  procedure  can  be  easily  extended  to  any  factorial 
design. 

The  approach  is  demonstrated  for  an  operational  test  involving  a 2^ 
factorial  design  and  the  results  are  compared  to  Classical”  procedures. 

The  sensitivity  of  the  required  input  parameters  is  investigated  and  related 
applications  are  discussed. 

The  proposed  approach  Is  found  to  be  a viable  method  of  designing, 
conducting,  and  evaluating  an  operational  test  Involving  a factorial 
experiment. 
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CHAPTER  I 

INTRODUCTION 

Background 

The  goal  of  any  type  of  experimentation  is  to  obtain  information 
about  the  system  under  investigation.  Naturally,  the  more  information 
that  can  be  obtained  the  better.  Unfortunately,  obtaining  more  informa- 
tion often  requires  increased  experimentation  and  the  resources  available 
may  become  a limiting  factor.  The  shortage  of  any  necessary  resource, 
whether  it  be  time,  money  or  materials  can  greatly  hinder  the  conduct  of 
the  experiment  and  even  preclude  obtaining  the  desired  amount  of  informa- 
tion from  the  experimentation. 

Much  effort  has  been  devoted  to  the  problem  of  how  to  best  utilize 
the  experimental  resources  available  in  order  to  gain  the  most  information 
from  them.  The  development  of  systematic  experiments  such  as.  factorial 
experiments  and  procedures  for  fractional  factorial  experiments  have  done 
much  to  improve  the  use  of  available  resources.  Screening  experiments 
have  aided  by  eliminating  needless  experimentation  involving  unimportant 
factors.  Also,  sequential  experiments  have  helped  to  obtain  the  desired 
Information  from  less  resources. 

The  problem  of  gaining  the  most  information  from  limited  resources 
can  be  found  everywhere.  From  agriculture  to  Industry  to  the  defense 
establishment,  everyone  is  interested  in  getting  the  most  for  their  money. 


f 


i 

I 


r 


2 


Material  Acquisition  Process 

The  requirement  to  maintain  a modern,  well-equipped  Army  requires 
constant  evaluation  and  reevaluation  of  existing  Army  equipment  to  insure 
that  it  is  adequate  to  fulfill  operational  requirements.  Based  on  the  ever 
changing  nature  of  these  requirements,  the  Army  is  involved  in  a continuous 
process  of  upgrading  its  current  equipment  and  procuring  new  items  of 
equipment.  Basically,  the  Army  satisfies  its  needs  for  new  equipment  in 
three  ways:  buying  equipment  already  developed,  evolutionary  development 
of  current  standard  equipment,  and  initiation  of  new  material  development 
programs.  All  of  these  procurement  methods  can  be  extremely  costly  in 
terms  of  time,  material,  and  money.  As  a result,  both  the  Department  of 
Defense  and  the  Department  of  the  Army  have  highly  structured  material  pro- 
curement policies  [7,  181  whose  objectives  are  to  minimize  the  costs  in- 
curred in  acquiring  material  systems  while  insuring  that  the  performance 
of  those  systems  is  adequate  to  meet  operational  requirements. 

Once  the  requirement  for  a new  or  updated  system  has  been  formalized, 
the  proposed  system  will  go  through  three  phases  of  development:  conceptual 
development,  validation,  and  full  scale  development,  before  the  system 
receives  approval  for  full  production  and  purchase  by  the  Department  of 
Defense. 

At  the  end  of  each  phase,  the  Defense  Systems  Acquisition  Review 
Council  (DSARC)  meets  to  provide  information  and  recommendations  to  the 
Secretary  of  Defense.  Based  on  these  recommendations,  the  Secretary  of 
Defense  may  decide  to  cancel  further  consideration  of  the  system,  require 
further  system  evaluation  prior  to  proceeding  on  to  the  next  stage,  or 
permit  the  system  to  pass  on  to  the  next  stage  of  development. 
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j At  Department  of  the  Army  level,  there  is  a similar  advisory  body, 

^ the  Army  Systems  Acquisition  Review  Council  (ASARC) , whose  principal  func- 
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tion  is  to  provide  the  DSARC  with  the  Army's  recommendations  concerning 
the  item  of  equipment  in  question. 

Testing 

To  aid  the  ASARC  in  its  recommendations,  testing  is  conducted  to 
I demonstrate  how  well  the  material  system  meets  its  technical  and  opera- 

' tional  requirements;  provide  data  to  assess  developmental  and  operational 

I 

f 

; risks  for  decision  making;  verify  that  the  technical,  operational,  and 

support  problems  identified  in  previous  testing  have  been  corrected;  and 

\ ' 

to  insure  that  all  critical  issues  to  be  resolved  by  testing  have  been 
adequately  considered.  Two  types  of  testing.  Developmental  Testing  (DT) 
and  Operational  Testing  (OT)  are  conducted.  DT  is  conducted  to  demon- 
strate that  the  engineering  design  and  development  process  is  complete, 

t 

that  design  risks  have  been  minimized,  and  that  the  system  will  meet 
required  specifications.  It  is  performed  by  the  material  developer  who 
then  forwards  the  results  to  the  ASARC. 

' OT  is  conducted  to  estimate  the  system's  military  utility,  opera- 

tional effectiveness,  operational  suitability,  and  the  need  for  any 
I modifications.  OT  can  also  provide  data  on  organization,  personnel 

requirements,  doctrine  and  tactics  for  the  new  system.  OT  is  performed 
under  the  supervision  of  the  U.S.  Army  Operational  Test  and  Evaluation 
Agency  (OTEA)  by  operational  and  support  personnel  of  the  type  and 
qualifications  expected  to  use  and  maintain  the  system  once  it  is 


' i 


deployed . 

As  a safeguard  and  as  a further  validation  measure,  DT  and  OT  test 
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designs  are  prepared  and  the  test  results  are  evaluated  independently. 

The  actual  testing  of  the  item  of  equipment  under  DT  and  OT  may,  however, 
be  conducted  concurrently  to  reduce  delays  and  system  acquisition  costs. 

The  relationship  between  the  conduct  of  the  different  phases  of  testing 
and  the  meeting  of  the  Acquisition  Review  Councils  is  shown  in  Figure  1. 

Operational  Testing 

OTF.A  serves  the  following  major  functions: 

A.  Insures  user  testing  is  effectively  planned,  conducted,  and  evaluated 
with  emphasis  on  adequacy,  quality,  and  credibility  of  all  user  testing. 

B.  Actively  participate  in  the  conduct  of  and  provide  independent  evalua- 
tions of  operational  tests  conducted  on  selected  items  of  equipment. 

C.  De\'elop  and  recommend  policy  on  user  testing. 

D.  Develop  and  promulgate  user  test  and  evaluation  methodology. 

E.  Develop  measures  of  effectiveness  and  provide  estimates  on  amount 

of  resources  (sample  size)  necessary  to  detect  differences  in  military 
utility,  operational  effectiveness,  and  operational  suitability  with 
a specified  confidence  level. 

Generally  three  phases  of  Operational  Testing  are  conducted,  one 
phase  prior  to  each  meeting  of  the  ASARC.  OT  will  compare  the  performance 
characteristics  of  the  new  system  against  the  current  system  if  it  is 
designed  as  a replacement  item,  against  a higher  level  system  if  it  is 
designed  as  a component  of  some  larger  system,  or  against  a set  of  perform- 
ance standards  if  the  item  is  totally  new  to  the  Army  Inventory.  Due  to 
the  nature  of  the  items  being  tested,  such  as  missile  systems  or  high  value 
Items  that  require  destructive  type  testing,  OTEA  is  often  limited  in  the 
number  of  experimental  runs  that  can  be  made  on  an  item  of  equipment.  As 
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a result,  OTEA  is  very  much  Interested  in  developing  a methodology  for 
designing,  planning,  and  evaluating  operational  tests  for  a limited 
sample  size  while,  at  the  same  time,  maximizing  the  information  gained 
from  the  test. 


Problem,  Objective,  Scope 

This  problem  was  motivated  by  a task  requirement  presented  by 


OTEA: 


Obtaining  maximum  information  from  minimum  sample  size  is  an  in- 
herent and  recurring  problem  in  operational  testing  and  has  signifi- 
cant impact  on  final  evaluations.  In  designing  and  conducting 
operational  tests,  resource  restrictions  often  dictate  a test  having 
an  extremely  small  sample  size  but  a number  of  influencing  factors 
of  two  or  more  conditions.  This  results  in  a relatively  large  num- 
ber of  combinations,  especially  considering  the  number  of  observa- 
tions to  be  obtained.  This  study  is  to  investigate  the  feasibility 
of  developing  a method  for  maximizing  the  information  gained  from 
a test. 

A related  problem,  that  of  designing,  planning,  and  evaluating 
operational  tests  of  limited  sample  size  has  already  been  addressed  by 
Russ  (48).  In  his  research,  he  developed  an  algorithm  for  determining 
the  optimum  constrained  sample  size  for  a full  factorial  experiment  based 
on  a specified  amount  of  information  required  by  the  test  evaluator. 

The  objective  of  this  study  will  be  to  develop  a sequential  method  of 
designing  and  conducting  an  operational  test  in  order  to  gain  equivalent 
information  from  an  even  smaller  sample  size. 

The  scope  of  this  research  will  be  limited  in  the  following  areas: 
A.  All  factors  in  the  factorial  design  will  appear  at  only  two  lev' Is 

and  will  be  considered  as  fixed  factors.  The  extension  of  the  problem 
to  cases  where  the  factors  appear  at  other  than  two  levels  will  pose 
no  problems  for  anyone  familiar  with  the  analysis  of  factorial 
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experiments. 

B.  Only  univariate  response  models  are  considered. 

C.  The  measured  response  is  assumed  to  come  from  a normal  population 

2 

with  mean,  u>  and  variance,  O . This  would  appear  to  be  a valid 
assumption  In  light  of  the  frequency  with  which  it  occurs  in  every- 
day situations. 

D.  The  hypotheses  to  be  tested  are  assumed  to  be  of  the  form  H^: 

vs.  > d.  This  assumption  is  based  on  the  fact  that  Opera- 

tional Testing  is  generally  performed  to  test  the  performance  charac- 
teristics of  one  system  against  another  system  or  against  a set  of 
standards.  The  decision  to  accept  or  reject  the  new  system  is  based 
on  whether  or  not  the  performance  characteristics  of  the  new  system 
exceed  the  old  system  by  a specified  margin,  d,  or  not. 

The  research  will  consist  of  a review  of  full  and  fractional  facto- 
rial design  construction  and  analysis  and  sequential  analysis  methodology. 
A proposed  sequential  method  for  building  a factorial  design  is  then 
developed  and  applied  to  a previously  performed  operational  test  to  demon- 
strate its  use  in  obtaining  the  same  Information  from  a reduced  sample 
size. 
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CHAPTER  II 

REVIEW  OF  OTHER  APPROACHES  AND  RELATED  TECHNIQUES 

This  chapter  will  present  a review  of  the  construction  and  analysis 
of  a factorial  experiment  and  will  review  other  approaches  to  the  sequen- 
tial analysis  problem.  Both  the  case  where  the  variance  is  known  and  the 
case  where  the  variance  is  unknown  are  examined  in  the  sequential  analy- 
sis review. 

Factorial  Experiments 

Full  Factorials 

A factorial  experiment  is  the  term  used  to  denote  the  experimental 
design  in  which  all  levels  of  a given  factor  are  combined  with  all  levels 
of  every  other  factor  in  the  experiment.  In  the  case  where  each  of  the 
n factors  can  be  measured  at  the  same  number  of  levels,  say  m,  the  experi- 
ment is  referred  to  as  an  m''  factorial.  For  example,  in  a 2^  factorial, 
each  of  the  n factors  would  appear  at  two  levels  so  there  are  2’^  differ- 
ent treatment  combinations.  In  a 2^,  the  levels  of  the  factors  are  arbi- 
trarily denoted  as  the  high  and  low  levels.  Generally  accepted  notation 
is  to  represent  the  various  factors  using  capital  letters.  A,  B,  C,  etc., 
and  to  denote  the  various  treatment  combinations  using  lower  case  letters, 
a,  b,  c,  etc.  The  letters  present  in  the  treatment  combination  indicate 
those  factors  appearing  at  their  high  level  and  the  letters  absent  indi- 
cate those  factors  appearing  at  their  low  level.  For  example,  the  treat- 
ment combination  a denotes  factor  A at  its  high  level  and  all  other 
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factors  at  their  low  levels  and  cd  denotes  factors  C and  D at  their  high 
levels  and  all  other  factors  present  in  the  experiment  at  their  low  levels. 
The  notation  (1)  is  used  to  represent  the  treatment  combination  where  all 
factors  appear  at  their  low  levels. 

The  common  representation  for  the  univariate  response  model  for 

2 

the  2 factorial  design  is: 


y...  = y + a.  + + a..  + e... 

^ijk  ^ i j ij  ijk 


(2-1) 


where 

y, = the  k observation  at  the  i level  of  factor  A and  the 
ijk 

level  of  factor  B. 

y = the  population  mean  of  the  observations. 

= effect  of  factor  A at  level  i,  1=1,2. 

= effect  of  factor  B at  level  j,  j=l,2. 

a. 6.  = interaction  effect  of  factor  A at  level  i and  factor  B at 
i J 

level  j . 

Ej = error  associated  with  observation  y . . 
ijk  -^ijk 

The  restriction  ZEnj.a_,  = °lj^j  ~ ^ where  n.  . number  of  observations 


in  the  ij^^  cell  also  applies.  This  model  can  easily  be  extended  to  n 
factors  by  the  addition  of  appropriate  terms. 

In  the  case  where  n^^  = 1 for  all  i and  j,  there  is  no  separate 
estimate  for  error  and,  in  general,  higher  order  interactions  are  assumed 
to  be  negligible  and  are  pooled  to  estimate  the  error.  The  most  desirable 
situation  is  the  case  where  there  is  more  than  one  observation  per  cell 


ij 


Ij  i 


ij 


and  where  the  number  of  observations  in  each  cell  is  the  same. 
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The  effect  due  to  a factor  Is  defined  to  be  the  change  in  the 
response  produced  by  a change  in  the  level  of  that  factor.  Interaction 
between  factors  exist  when  a change  in  one  factor  produces  a different 
change  in  the  response  variable  at  one  level  of  another  factor  than  at 
the  other  level  of  that  factor. 

All  of  the  treatment  combinations  present  in  a factorial  experiment 
can  be  expressed  in  standard  order.  The  simplest  method  of  writing  the 
treatment  combinations  in  standard  order  is  the  method  of  signs  as  demon- 
strated in  Montgomery  (42).  This  method  consists  of  listing  all  n factors 
as  column  headings.  The  first  column  consists  of  a total  of  2*^  alternat- 
ing minus  and  plus  signs  starting  with  a minus  sign.  The  second  column 
consists  of  alternating  pairs  of  minus  and  plus  signs  starting  with  two 
minus  signs.  The  third  column  consists  of  alternating  sets  of  four  minus 
and  plus  signs.  In  general,  the  column  for  the  n^^  factor  will  consist 
of  alternating  sets  of  2^  ^ minus  and  plus  signs  always  beginning  with 
a set  of  minus  signs  until  there  are  a total  of  2^  entries  in  that  column. 
The  combinations  of  minus  and  plus  signs  in  each  row  are  the  treatment 
combination  represented  by  that  row  where  the  minus  sign  indicates  a 

factor  at  its  low  level  and  a plus  sign  indicates  a factor  at  its  high 

4 

level.  This  procedure  is  Illustrated  for  a 2 factorial  experiment  in 
Table  1. 

Blocking 

In  experimental  designs  Involving  a large  number  of  trials,  it  may 
be  impossible  to  perform  the  entire  experiment  under  homogeneous 
conditions.  In  this  case,  the  experimenter  may  have  to  perform  the 
experiment  in  blocks.  In  order  to  separate  the  full  factorial  Into 
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Table  1.  Method  of 

Signs  for  a 

4 

2 Factorial 

Treatment 

Factor  A 

Factor  B 

Factor  C 

Factor  D 

Combination 

- 

- 

- 

- 

(1) 

+ 

- 

- 

- 

a 

- 

+ 

- 

- 

b 

+ 

+ 

- 

- 

ab 

- 

- 

+ 

- 

c 

+ 

- 

+ 

- 

ac 

- 

+ 

+ 

- 

be 

+ 

+ 

+ 

- 

abc 

- 

- 

- 

+ 

d 

+ 

- 

- 

+ 

ad 

- 

+ 

- 

+ 

bd 

+ 

+ 

- 

+ 

abd 

- 

- 

+ 

+ 

cd 

+ 

- 

+ 

+ 

acd 

- 

+ 

+ 

+ 

bed 

+ 

+ 

+ 

+ 

abed 

12 


blocks,  the  experimenter  must  decide  which  effect  (s)  he  is  not  interested 
in  or  can  assume  to  be  negligible.  Usually,  this  is  the  highest  order 
interaction  or  Interactions  present.  Once  this  decision  is  made,  a defin- 
ing contrast,  an  expression  stating  which  effects  are  to  be  confounded 
(not  estimable)  with  blocks,  is  established.  Hicks  [25]  and  Kempthorne 
[36]  demonstrate  several  methods  for  separating  the  treatment  combinations 
into  blocks.  Each  block  should  contain  a number  of  treatment  combinations 
no  greater  than  the  number  of  experimental  trials  that  can  be  performed 
at  on**  time. 

Regular  Fractional  Factorials 

In  the  case  of  a large  number  of  treatment  combinations  present  in 
an  experiment,  it  may  not  be  either  physically  or  economically  possible 

for  the  experimenter  to  obtain  an  observation  at  every  treatment  combl- 

ii 

j nation.  However,  it  is  possible  to  run  only  a fraction  of  the  experiments 

1. 

[ and  still  obtain  the  same  information  as  if  the  entire  experiment  had  been 

performed.  The  general  procedure  is  known  as  fractional  replication. 

Box  and  Hunter  [29]  list  the  following  as  major  uses  of  fractional 
factorial  designs: 

A.  When  certain  interactions  can  be  assumed  non-significant  from  prior 
knowledge,  these  Interactions  can  be  used  as  generators  in  the  sepa- 
ration of  treatment  combinations  into  their  respective  fractions  in 
such  a way  as  to  make  efficient  use  of  the  analysis  of  the  fractional 
design. 

I B.  In  screening  experiments  where  it  is  expected  that  the  effects  of  all 

1 
I 

I 


but  a few  of  the  variables  studied  will  be  negligible,  the  use  of 
fractional  factorials  as  a screening  experiment  will  enable  the 
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1 


I 

i 

experimenter  to  determine  which  factors  are  Important  and  which  are 
not  without  performing  costly  and  unnecessary  experimentation.  The 
insignificant  factors  may  then  be  set  at  standard  levels  and  the 
significant  factors  explored  more  comprehensively  during  further 

experimentation  using  a smaller,  less  costly  factorial  design.  j 

c.  Where  groups  of  experiments  are  run  in  sequence  and  ambiguities  j 

remaining  at  a given  stage  of  experimentation  can  be  resolved  by 

later  groups  of  experiments,  the  full  factorial  can  be  fractionated 
so  that  the  fractional  experiment  performed  yields  Information  on 
only  the  factors  in  question. 

d.  Where  certain  major  variables,  which  may  interact,  are  to  be  studied 
simultaneously  with  other  minor  variables  whose  influence,  if  any, 
can  be  described  by  main  effects  only,  the  fractional  design  can  be 
established  so  as  to  confound  the  major  variable  main  effects  with 
only  two  factor  interactions  of  the  minor  variables  but  these  inter- 
actions are  assumed  to  be  negligible  so  they  have  no  effect  on  the 
main  factors. 

In  order  to  separate  the  full  factorial  into  fractions,  it  is 
necessary  for  the  experimenter  to  determine  which  effects  he  wishes  an 
estimate  for  and  which  effects  he  is  willing  to  assume  as  negligible. 

Fractional  factorial  designs  can  be  classified  in  the  following  manner 
for  convenience: 

a.  Resolution  II  designs:  main  effects  are  not  confounded  with  each 

other  but  are  confounded  with  two  factor  interactions  and  two  factor 

interactions  are  confounded  with  each  other.  The  smallest  possible 

3-1 


resolution  III  design  is  the  2 


design  which  can  be  generated 
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I 


using  either  1 = ABC  or  I = -ABC  as  the  defining  relationship.  This 

design  can  be  easily  generated  by  the  method  of  signs.  First,  a full 
2 

2 design  using  - and  + signs  is  written  down.  The  signs  in  the 
third  column  are  generated  as  the  product  of  the  signs  in  the  first 
two  columns  multiplied  by  +1  if  I = ABC  is  the  generator  used  and  by 
-1  if  I = -ABC  is  the  generator  used.  This  is  equivalent  to  equating 
the  third  factor,  c,  to  the  product  of  the  first  two  (C  = AB)  or  to 
the  negative  product  of  the  first  two  (C  = -AB) . This  procedure  is 
illustrated  in  Table  2. 

3-1 

Table  2.  2 Resolution  III  Design 


2 

Full  2 

A 

B 

i 

+ 

1 

- 

+ 

1 

+ 

+ 

1 

\ 

Treatment 

Treatment  I 

A 

B 

C 

Combination 

A 

B 

C 

Combination  j 

- 

- 

+ 

c 

- 

- 

- 

(1) 

+ 

- 

- 

a 

+ 

- 

+ 

ac 

- 

+ 

- 

b 

- 

+ 

+ 

be 

+ 

+ 

+ 

abc 

+ 

+ 

- 

ab  1 

b.  Resolution  IV  designs:  main  effects  are  not  confounded  with  each 
other  or  with  two  factor  Interactions  but  two  factor  Interjictions 
are  confounded  with  each  other.  The  smallest  resolution  IV  design 


15 


4-1 

is  the  2 which  can  be  generated  in  the  same  manner  as  in  the  previous 

3 6“2 

example  by  starting  with  a full  2 . A 2 Resolution  IV  design  with 
I “ ABCE  and  I = ABDF  as  the  generators  is  shown  in  Table  3. 


Table  3. 

^6-2  ^ ^ 

2iv  Design, 

I = ABCE 

and  I = 

ABDF 

Factor 

Factor 

Factor 

Factor 

Treatment 

A 

B 

C 

D 

E=ABC 

F+ABD 

Combination 

- 

- 

- 

- 

- 

- 

(1) 

+ 

- 

- 

- 

+ 

+ 

aef 

- 

+ 

- 

- 

+ 

+ 

bef 

+ 

+ 

- 

- 

- 

- 

ab 

- 

- 

+ 

- 

+ 

- 

ce 

+ 

- 

+ 

- 

- 

acf 

- 

+ 

+ 

- 

- 

+ 

bcf 

+ 

+ 

+ 

- 

+ 

- 

abce 

- 

- 

- 

+ 

- 

+ 

df 

+ 

- 

- 

+ 

+ 

- 

ade 

- 

- 

+ 

+ 

- 

bde 

+ 

+ 

- 

+ 

- 

+ 

abdf 

- 

- 

+ 

+ 

+ 

+ 

cdef 

+ 

- 

+ 

+ 

- 

- 

acd 

- 

+ 

+ 

+ 

- 

- 

bed 

+ 

+ 

+ 

+ 

+ 

+ 

abedef 
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c.  Resolution  V designs:  no  main  effects  or  two  factor  interaction  is 
confounded  with  any  other  main  effect  or  two  factor  interaction,  but 
two  factor  Interactions  are  confounded  with  three  factor  interactions. 
The  smallest  Resolution  V design  is  the  2^  ^ which  can  also  be  con- 
structed in  the  same  manner  as  the  Resolution  Til  and  IV  designs 
previously  described. 

In  general,  the  resolution  of  a design  is  equal  to  the  smallest 
number  of  characters  in  any  word  appearing  in  the  defining  relation.  The 
words  in  the  defining  relation  consist  of  the  generators  initially  chosen 
and  all  of  their  generalized  interactions  (products  mod  2 on  exponents). 
For  the  design  shown  in  Table  3,  the  complete  defining  relation  is 
I = ABCE  = ABDF  = CDEF. 

One  problem  that  will  arise  in  fractionating  a full  factorial  is 

that  two  or  more  effects  may  have  the  same  numerical  value.  In  this  case, 

the  effects  are  known  as  aliases  and  the  experimenter  must  be  sure  that 

factors  believed  to  be  significant  are  not  aliased  with  each  other.  The 

aliases  of  any  factor  can  be  generated  by  multiplying  that  factor  by  all 

of  the  words  in  the  defining  relation,  mod  2 on  exponents.  The  alias 
6—2 

structure  for  the  2 Resolution  IV  design  with  defining  relation 

I = ABCE  = ABDF  = CDEF  is  shown  in  Table  4.  For  a 2^  ^ fractional  fac- 

p 

torial  design,  each  effect  will  have  2 -1  aliases. 

When  estimating  the  effect  of  a factor  for  a fractional  factorial 
design,  we  are  really  estimating  the  effect  due  to  that  factor  and  all 
of  its  aliases.  Therefore,  the  effect  of  A is  really  a measure  of  the 
effect  due  to  A + BCE  + BDF  + ACDEF.  If,  as  in  the  general  case,  inter- 
actions of  third  order  or  higher  are  assumed  to  be  negligible,  then 


Table  4.  Alias 

I = 

Structure  for 

ABCE  = ABDF  = 

2^"^  Design, 

CDEF 

I 

ABCE 

ABDF 

CDEF 

A 

BCE 

BDF 

ACDEF 

B 

ACE 

ADF 

BCDEF 

C 

ABE 

ABCDF 

DEF 

D 

ABCDE 

ABF 

CEF 

E 

ABC 

ABDEF 

CDF 

F 

ABCEF 

ABD 

CDE 

AB 

CE 

DF 

ABCDEF 

AC 

BE 

BCDF 

ADEF 

AD 

BCDE 

BF 

ACEF 

AE 

BC 

BDEF 

ACDF 

AF 

BCEF 

BD 

ACDE 

CD 

ABDE 

ABCF 

EF 

CF 

ABEF 

ABCD 

DE 

ACD 

BDE 

BCF 

AEF 

ADE 

BCD 

BEF 

ACF 

this  response  will  provide  a good  estimate  of  the  effect  due  to  factor  A. 
The  analysis  of  this  effect  will  be  discussed  in  a later  section. 

If,  as  in  the  case  of  a large  number  of  factors  present  in  the 
experiment,  a one-half  fraction  still  leaves  too  many  observations  to  be 
taken,  it  is  possible  to  use  a smaller  fraction.  To  run  a 2*^  ^ frac- 


tional factorial  design  requires  the  choice  of  P independent  generators 


(no  ctiosj’n  nr  Is  a ni'iU'raHzoil  (ntoraotlon  nt  t Un  ntlinrs). 

Irrojjular  Fract  tonal  Kaolor  ta  Is 

In  many  cast's,  especially  in  an  experiment  Involving  a larne  nnm- 
her  ot  variables,  t lie  rnnnlny,  iit  a reunlar  traction  of  a 2*',  a 2*'  * say, 
may  luit  be  economical.  For  example,  suppose  the  experiment  In  qiu'st  Ion 
Is  a 2^  anil  t be  experimenter  can  al'lonl  to  pertorin  10  trials  ol'  the 
experiment.  If  lie  wishes  clear  estimates  ol  all  main  ellects  aiul  two  lac- 
tor  Interaction,  a 1/2  replicate  ol  the  2^  will  provlile  t lit'm  but  It  re- 
ipilres  64  trials,  more  than  the  allotteil  number.  A 1/4  replicate  ol  the 
2^  reipilres  only  12  trials  but  "there  iloes  not  exist  a 1/4  rei'llcate  ot 
the  2^  experiment  which  allows  niicorre lat ed  estimates  ol  all  main  ellects 
and  two  factor  inti'ract  Ions. " (1)  The  loRic.al  quest  Ion  that  arises  is 

whether  or  not  a pi. 'in  can  be  constructed  uslny,  close  to  but  not  more  than 
lO  trials  thill  will  yield  clciir  esiimati's  of  the  iiiiiin  ellects  ;ind  two 
fiictor  interiict  Ions.  Addelniiin  |l|  and  ,lohn  (Tl  | both  propose  slip.hllv 
different  solutions  to  the  development  of  this  design. 

1> 

Addi'lniiin  defines  his  irreunliir  traction  iis  ii  K/2  lr;ictlon  ol  ii 
2"  fiictorliil.  He  builds  his  irrep.nliir  tfiicllon  bv  combining,  the  treiilment 
comb  1 Hilt  1 ons  in  K distinct  1/2  repllciiles  ol  the  2 . In  Addelm;in's 
Irreunliir  fraction  pl.'ii,  no  miiln  ellect  or  lnter;iclion  need  bi'  completely 
confounded  with  the  me;in  11  K • 1’  f 1 but  II  K " 1’  - |i,  |i  " (1,  1,  2...., 
then  n f 1 efiecis  or  Inti'fiict  ions  iiiul  their  neneiii  I I r.ed  Ini  er;icl  Ions 
will  be  completely  conlonnded  with  the  meiin.  II  Ibis  should  hiipiu'n. 
however,  most  ol  the  lime  It  will  be  possible  to  construct  the  Irrep.uliir 
ftiicllon  so  that  the  etlects  which  itri’  compli'lelv  conlonnded  with  the 
meiin  will  contiiln  ill  leiist  live  liiclors.  In  the  insliinci's  where  this 
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irregular  fraction  cannot  be  constructed,  some  two  factor  interactions 
will  not  be  estimable.  Addelman  shows  that  the  yield  of  the  treatment 
combinations  in  his  Irregular  fraction  can  be  expressed  in  terms  of  the 
main  effects  and  two  factor  Interactions  in  the  following  manner: 

= M 1 '5  A ± S B 1 ‘2  AB  ♦ '2  C + 'i  AC,  etc.  + error  (2.2) 

where  the  sign 

on  A is  - if  i=0  and  + if  1=1 

on  B is  - if  )=0  and  + if  j=l 

on  AB  is  - if  the  product  of  the  signs  on  A and  B is  - and  is  + 

if  the  product  of  the  signs  on  A and  B is  +,  and  so  on. 

In  this  case,  0 and  1 would  indicate  the  presence  of  that  factor  at  its 
low  and  high  levels  in  the  treatment  combination  in  question.  In  the 
appendix  to  his  paper,  Addelman  gives  several  common  and  useful  irregular 
fractions  with  the  identity  relationship  and  assumptions  required  to 
generate  them.  One  such  example  is  the  3/8  fraction  of  the  2^  which 

would  be  useful  in  the  situation  described  at  the  beginning  of  this 

section.  Using  the  identity  relationship  I = ABODE  = ABF  = CDEF  = AEC  = 
BCDG  = BEFG  = ACDFG,  the  2^  can  be  fractionated  into  eight  blocks  of  16 
runs  each.  The  experimenter  could  then  pick  three  of  these  blocks, 
yielding  48  experimental  trials. 

•Tohn  approaches  the  problem  of  Irregular  fractions  by  subtracting 
treatment  combinations  from  the  full  factorial  or  by  adding  treatment 
combinations  to  a 1/2  replicate  of  the  full  factorial.  He  defines  ills 
designs  in  terms  of  the  relationship  used  to  generate  the  missing  fraction. 


dohn  illustrates  how  the  combination  of  fractions  In  a certain  manner 
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will  form  overlapping  fractions.  Estimates  for  the  effects  due  to  vari- 
ous treatment  combinations  are  then  obtained  by  combining  the  estimates 
obtained  from  the  appropriate  overlapping  fractions. 

Analysis  of  Factorial  Experiments 

Once  the  experiment  has  been  performed,  the  results  must  be  ana- 
lyzed to  determine  the  significance  or  insignificance  of  the  various 
factors  and  interactions  or  to  perform  tests  on  the  various  hypotheses 
concerning  the  factors.  The  most  commonly  used  method  of  analysis  is 
the  analysis  of  variance  (ANOVA) . 

ANOVA  makes  use  of  the  fact  that  the  ratio  of  two  chi-square  ran- 
dom variables  divided  by  their  respective  degrees  of  freedom  follows  an 
F distribution.  Hicks  [25]  and  Hines  and  Montgomery  [26]  provide  the 
general  format  for  an  ANOVA  table  for  a 2^^  factorial  experiment  with  r 
replications  per  cell  shown  in  Table  5.  The  statistic  is  found  by 

forming  the  ratio  of  ms  for  the  effect  in  question  to  ms  . Comparing 

h 

this  value  to  the  value  of  the  F statistic  with  1 and  2*^(r-l)  degrees 
of  freedom  will  give  a test  on  the  significance  of  an  effect. 

The  easiest  method  for  computing  the  sum  of  squares  for  a 2*^  fac- 
torial is  the  Yates  Method  [57]  which  consists  of  arranging  the  treatment 
combinations  in  standard  order  and  then  adding  and  subtracting  the 
observed  response  values  in  pairs  a total  of  n times  to  obtain  an  esti- 
mate of  the  contrast  due  to  a treatment  combination.  The  effect  due  to 
that  treatment  combination  is  then  obtained  from 

EFFECT  = (2) (CONTRAST) /r. 2"  (2.3) 


where  r 


number  of  replications  per  cell.  This  will  usually  be  one  for 


v, 

t 

i 
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Table  5.  ANOVA  for  2^  with  r Replications  Per  Cell 


Source  of  Variation 

Sum  of 
Squares 

Degrees  of  Freedom 

Mean 

Square 

Main  Effects — A 

S^A 

1 

^^A/1 

B 

^^B 

1 

^Vl 

n 

C 

1 

s^c/i 

2 Factor  Interactions — AB 

^^AB 

1 

^^AB/1 

AC 

^^AC 

1 

^^AC/1 

c(n,2)  - 

BC 

S^AC 

1 

^^AC/1 

n(n-l) 

. . . 

2 

3 Factor  Interactions — ABC 

^^BC 

1 

ABD 

^^ABD 

1 c(n,3)  = 

BCD 

^^BCD 

1 n (n-1) (n-2) 

6 

4 Factor  Interactions,  Etc. 

Sum  of  all  Treatment 

Combinations 

SS 

2^-1 

ss„ 

Residual  or  Error 

ss”^ 

2”(r-l) 

E 

2"(r-l) 

Total 

SSt 

r(2")-l 
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purposes  of  this  paper.  The  sum  of  squares  for  any  effect  Is  then 
obtained  from 

SS  = (CONTRAST) ^/r- 2"  (2.4) 

Because  of  Its  simplicity,  the  Yates  method  is  easily  programmed  for 
computer  application. 

The  Yates  method  can  also  be  used  in  obtaining  the  sums  of  squares 

n— p 

from  a 2 fractional  factorial  by  considering  the  data  as  having  come 

from  a full  factorial  in  n-P  variables.  The  treatment  combinations  for 

the  full  factorial  are  written  in  standard  order  and  a letter  or  letters 

is  added  in  parentheses  to  the  end  of  these  treatment  combinations  to 

represent  the  treatment  combinations  actually  run.  The  effect  estimated 

in  this  manner  will  then  be  the  effect  associated  with  the  treatment 

combination  shown  plus  all  of  its  aliases.  An  example  of  the  Yates 
4-1 

method  for  a 2 with  I = ABCD  as  the  defining  relationship  is  shown  in 
Table  6. 

The  values  in  column  (1)  are  obtained  by  first  adding  the  respon- 
ses in  pairs  and  then  subtracting  them  in  pairs.  For  example, 

82  = 74  + 108,  222  = 92  + 130,  173  = 68  + 105,  228  = 95  + 133  and 
34  = 108  - 74,  38  = 130  - 92,  37  = 105  - 68  and  38  = 133  - 95.  The  values 
in  columns  (2)  and  (3)  are  obtained  by  performing  the  same  operations 
on  the  values  in  columns  (1)  and  (2)  respectively.  It  is  necessary  to 
perform  these  operations  three  times  since,  in  this  case,  n - P = 

4-1  = 3. 
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Table  6. 


Yates  Algorithm  for 


2^  ^ with  I = ABCD 


[42] 


Treatment 

Combination 

Response 

(1) 

(2) 

(1) 

74 

182 

404 

a(d) 

108 

222 

401 

b(d) 

92 

173 

72 

ab 

130 

228 

75 

c(d) 

68 

34 

40 

ac 

105 

38 

55 

be 

95 

37 

4 

abc (d) 

133 

38 

1 

(3) 

Effect 

Estimate 
of  Efftct 
2x(3)/2" 

SS 

(3)2/20 

805 

- 

147 

A+BCD 

36.75 

2701.125 

95 

B+ACD 

23.75 

1128.125 

5 

AB+CD 

1.25 

3.125 

-3 

C+ABD 

-0,75 

1.125 

3 

AC+BD 

0.75 

1.125 

15 

BC+AD 

3.75 

28.125 

-3 

ABC+D 

-0.75 

1.125 

Sequential  Procedures 


Introduction 

Until  recently,  commonly  accepted  statistical  procedures  involved 
presenting  the  data  from  an  already  conducted  experiment  to  the  statisti- 
cian and  expecting  him  to  provide  reasonable  conclusions  based  on  an 
analysis  of  the  data.  Seldom  was  the  statistician  consulted  concerning 
the  methods  of  collecting  the  data  either  prior  to  or  during  the 
experimentation.  Lately,  however,  much  more  emphasis  has  been  placed 
on  obtaining  the  advice  of  a statistician  prior  to  the  performance  of 
an  experiment  and  during  the  actual  running  of  the  experiment.  This  has 
allowed  the  statistician  to  play  an  Important  role  in  designing  and  moni- 
toring the  conduct  of  experiments.  In  addition,  more  emphasis  has  been 


J 
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placed  on  obtaining  meaningful  results  from  a reduced  number  of  trials 
of  an  experiment  due  to  the  rapidly  increasing  costs  associated  with 
experimentation.  Fractionating  a factorial  design  is  one  way  of  accom- 
plishing this  goal.  A second  useful  method  is  through  sequential  analy- 
sis procedures. 

Sequential  analysis  can  best  be  described  as  "any  statistical 
procedure  in  which  the  final  pattern  (including  the  number  ) of  observa- 
tions is  not  determined  a priori  but  depends,  in  some  way  or  other,  on 
the  values  observed  in  the  course  of  the  work"  [34].  During  the  course 
of  an  experiment  testing  one  hypothesis  against  another,  the  results  are 
examined  after  each  trial.  A decision  is  then  made  to  either  accept 
one  or  the  other  of  the  hypotheses  or  to  continue  sampling  based  on  the 
results  to  date.  The  decision  making  process  of  sequential  analysis  will 
require  three  basic  rules  to  define  the  procedure: 

A.  The  stopping  rule  will  let  the  experimenter  know  when  the  experimen- 
tation may  be  terminated. 

B.  The  terminal  decision  rule  will  let  the  experimenter  know  which 
hypothesis  to  decide  in  favor  of  once  experimentation  has  been 
terminated. 

C.  The  experimentation  rule  will  let  the  experimenter  known  which  experi- 
ment should  be  performed  next  should  he  still  be  required  to  continue 
expe  r Imentat ion . 

Curtailed  Sampling 

Although  Wald  [52]  is  generally  credited  with  originating  sequen- 
tial analysis  procedures  in  the  mid  1940' s,  there  were  some  heuristic 
approaches  made  to  this  area  prior  to  Wald.  One  such  approach  is  known 
as  curtailed  sampling  and  is  used  in  determining  whether  to  accept  or 
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as  curtailed  sampling  and  is  used  in  determining  whether  to  accept  or 
reject  a lot  of  some  item.  "Classical"  sampling  procedures  would  call 
for  a fixed  sample  of  size  n to  be  inspected  and  the  decision  to  accept 
or  reject  the  entire  lot  would  be  based  on  the  number  of  defective  items 
found  in  that  sample.  Curtailed  sampling  procedures  would  stop  sampling 
as  soon  as  it  was  obvious  which  decision  would  have  to  be  made.  For 
example,  if  the  fixed  size  test  called  for  rejecting  a lot  if  more  than 
two  defective  items  were  found  in  a sample  of  size  ten,  curtailed  samp- 
ling would  call  for  termination  of  sampling  as  soon  as: 

A.  Three  defective  items  were  found, 

B.  Nine  items  had  been  inspected  and  only  one  defective  item  was  found,  or 

C.  Eight  items  had  been  inspected  and  no  defective  items  were  found. 

Tests  on  the  Mean  of  a Normal  Distribution  with  Known  Variance 

Sequential  Probability  Ratio  Test.  Wald's  Sequential  Probability 

Ratio  Test  (SPRT)  is  designed  to  test  one  simple  hypothesis  against 

another.  If  a random  variable,  x,  has  distribution  f(x,0),  the  SPRT  will 

test  HqIO  = 6q  vs.  0 = 0^  and  will  decide  in  favor  of  either  or 

H,  based  on  a series  of  observations  of  x(x,  , x.,,  ...,  x ) and  on  pre- 
1 1 2 n 

selected  probabilities  of  type  I (reject  Hq  when  it  is  true)  and  type  II 
(accept  Hq  when  it  is  false)  errors.  If  these  probabilities  are  denoted 
as  a and  6 respectively,  there  are  four  possible  outcomes  as  shown  in 
Table  7. 

For  a fixed  sample  size,  n,  Neyman  and  Pearson,  as  explained  in 
Hoel,  Port  and  Stone  [28],  have  shown  that  the  most  powerful  test  (that 
test  giving  the  smallest  B)  depends  on  the  likelihood  ratio: 
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Li(x) 


n f(x^,0j) 


L„(x)  n 

" n f(x.,e„) 

i=l  ^ ^ 


(2.5) 


where  f(x^,  0^)  = probability  of  observing  x^  given  that  0^  Is  true. 
This  test  will  decide  In  favor  of  if  Lj^(x)/Lq(x)  is  less  than  some 
constant,  k,  and  will  decide  against  if  L^(x)/Lq(x)  is  greater  than 
k.  The  value  of  k can  be  chosen  to  Insure  the  desired  a value  for  the 
test  and  the  number  of  observations,  n,  can  be  chosen  to  insure  the 
desired  power,  1-6,  for  the  test. 


Table  7.  Outcomes  and  Probabilities  of  SPRT 


Required 

Outcome Probability 


% 

Is 

true 

and 

test 

decides 

in 

favor 

of 

1 

-a 

% 

is 

true 

and 

test 

decides 

in 

favor 

of 

®i 

a 

®1 

Is 

true 

and 

test 

dec  ides 

in 

favor 

of 

®i 

1 

-6 

®1 

Is 

true 

and 

test 

decides 

in 

favor 

of 

6 

The  SPRT  is  similar  to  this  procedure  and  incorporates  the  follow- 
ing decision  rules: 

A.  Stop  sampling  and  decide  in  favor  of  as  soon  as  Lj^(x)/Lq(x)  is 
less  than  some  constant,  B. 

B.  Stop  sampling  and  decide  in  favor  of  as  soon  as  Lj(x)/Lq(x)  is 


greater  than  some  constant,  A > B. 
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C.  Continue  sampling  as  long  as  B < L^(x)/Lq(x)  < A. 

Wald  [52]  has  proven  that  good  approximations  for  the  constants  A 
and  B are: 


A = (l-3)/a 

(2.6) 

B = B/(l-a) 

(2.7) 

Since  a and  3 are  generally  taken  to  be  less  than  .5,  the  following 
relationship  will  hold: 

0 < B < 1 < A (2.8) 

The  Sequential  Probability  Ratio  Test  possesses  many  properties 
that  make  it  very  useful  when  testing  a simple  hypothesis  against  a simple 
alternative. 

A.  Wald  and  Wolfowltz  [53]  have  shown  that  for  all  sequential  tests 
having  the  same  a and  3 probabilities  of  Type  1 and  Type  II  error,  the 
SPRT  will  require  the  fewest  number  of  observations  when  0 has  true 
value  equal  to  0^  or  0^^. 

B.  Should  the  true  value  of  0 lie  somewhere  between  Pq  and  0^^,  Wether  ill 
[56]  has  shown  that  the  expected  number  of  observations  may  be  much 
larger  than  the  fixed  sample  size  plan  with  the  same  a and  3 errors. 

C.  Wald  has  proposed  truncating  the  SPRT  at  some  fixed  value,  n^,  should 
this  occur.  This  will  change  the  probabilities  of  Type  I and  Type  II 
errors  so  the  problem  is  to  make  n^  large  enough  so  as  to  have  a 
negligible  effect  on  these  probabilities.  By  denoting 
probability  of  a sample  of  size  n^  rejecting  under  the  truncated 
process  and  accepting  under  the  non-t runcated  process,  Wald  derives 
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upper  bounds  for  these  two  probabilities  and  uses  them  to  determine 
upper  bounds  for  the  new  a and  B errors  respectively.  The  upper 


bounds  for  P^Cn^)  and  ^^e 


= 0(02)  - G(uj^)  and  = G(u^)  - G(u2) 


(2.9) 


where 


/n~  OqCz) 


“2  = 


log  A-nQEQ(z) 

Oq(z) 


log  B-n^E^Cz) 

/n^  Oj^(z) 


-noEi(z) 


(2.10) 


^(z) 


G(u)  = probability  that  a N(0,1)  variable  takes  on  a value  < u. 
o^(z)  = standard  deviation  of  z when  is  true  i - 0,1 
Eo(z)  = - |(9q  - 0^)^  E^(z)  = -1(0^  - e^)^ 

The  value  of  n^  must  be  sufficiently  large  to  insure  z^  + . . . + ~ N 

with  mean  equal  to  nQE^(z)  and  standard  deviation  equal  to  /n^  o^(z) 
when  is  true,  i = 0,1.  The  upper  bounds  for  the  a and  3 errors  are 
then  given  by: 


«(V  - “ Po^"o^ 


3(nQ)  £ 6 + Pj(nQ) 


Modifications  to  SPRT 


(2.11) 

(2.12) 


As  was  stated  earlier,  Wald's  SPRT  will  require  a smaller  average 
number  of  observations  than  any  other  sequential  test  of  one  simple  hypo- 
thesis against  another  when  the  true  value  of  0 is  either  0^  or  0^^.  This, 
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Is  not  always  the  case.  Read  [47]  and  Anderson  [2]  have  proposed  modifi- 
cations to  the  SPRT  in  case  the  true  value  of  0 lies  between  0^  and  0^. 

Read's  Partial  Sequential  Probability  Ratio  Test  (PSPRT)  uses  the 
same  Initial  boundaries,  A and  B,  as  Wald's  test.  Initially,  a fixed 
number,  n,  of  observations  is  taken.  At  the  end  of  these  n observations, 
the  quantities: 

B’  = B — (2.13) 

n p n 


and 


A' 

n 


V 

Pin 


(2.14) 


where  p^^  = joint  likelihood  of  the  first  n observations  under  h^,  i=0,l 
are  computed.  If  the  inequality 


0 < B'  < 1 < A'  (2.15) 

n n 

does  not  hold,  a decision  is  made  at  this  point.  If  A^  < 1,  then  is 

accepted  and  if  B'  >1  then  H.  is  accepted.  If  (2.15)  holds,  then 
n 0 

sampling  is  continued  as  in  the  SPRT  and  the  following  decision  rules 
are  used: 

A.  Stop  sampling  and  decide  in  favor  of  as  soon  as 

P/^(^  ,1  »•••>’'  ?)  becomes  less  than  B' . 

0 n+l  n n 

B.  Stop  sampling  and  decide  in  favor  of  as  soon  as  Pj^  (^n+l ’ ’ ’ ' ’ ^n' ^ ^ 

Po(^  i)  becomes  greater  than  A'. 

*^0  n+l  n ° n 

C.  Continue  sampling  as  long  as  B^  < Pj^^^^n+l* ' ' ’ ’’^n' ^^^O^’^n+l’ ' ' " ’^n' ^ 

< A'  . 

n 

n+l 


Where  p.  (x  .•.,x  ,)  is  the  joint  likelihood  of  observations  x 
i n+l,  n : 


to 
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^'>n  under  H^,  i = 0,1. 

Read  shows  that  his  PSPRT  will  require,  on  the  average,  more  obser- 
vations than  the  SPRT  when  the  true  value  of  9 is  9^  or  9^  but  may  require 
substantially  fewer  observations  when  the  true  value  of  9 is  between  9^ 
and  9j^,  say  9 = *5(9^  + both  cases.  Read's  Average  Sample  Number 

(ASN)  is  lower  than  the  fixed  sample  size.  Read's  procedure  is  most  use- 
ful when  it  is  desirable  to  take  n initial  observations  for  some  reason 
such  as  to  provide  the  experimenter  with  an  estimate  of  the  sampling  costs. 

Anderson's  modification  to  the  SPRT  applies  only  to  densities  of 
the  Koopman-Darmois  (exponential)  form  and  he  presents  specific  examples 
for  a normal  distribution  with  unknown  mean  and  known  variance.  Since 
many  random  variables  follow  a normal  distribution,  or  do  so  approximately, 
this  is  a reasonable  approach. 

In  the  SPRT,  the  continuation  region  boundaries,  A and  B,  can  be 
thought  of  as  describing  two  parallel  lines,  y = A and  y = B,  where  the 
decision  is  made  as  soon  as  the  value  of  y,  in  this  case  the  likelihood 
ratio,  crosses  one  of  the  lines.  Anderson  proposes  replacing  the  paral- 
lel lines  with  a set  of  converging  lines  y = c^  + d^^n  and  y = C2  + d^n 
with  truncation  of  the  sequential  procedure  at  some  value  N.  To  avoid 
intersection  of  the  lines  before  n = N,  it  is  necessary  that  C2  + d^CN-l) 

< Cj  + dj^(N-l).  It  is  also  desirable  that  the  lines  converge,  so  it  is 
necessary  that  d^^  < 0 < d2. 

For  a normal  distribution,  as  described  here,  Hoel,  Part  and  Stone 

[28]  have  shown  that  the  likelihood  ratio  can  be  replaced  by  the  quantity 
n 

X with  appropriate  modifications  to  the  boundaries  of  the  continuation 
i = l ' 

region.  Making  this  substitution,  the  decision  process  for  Anderson's 
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1 


SPRT  becomes: 

A.  Stop  sampling  and  decide  in  favor  of  as  soon  as 


n 


I X.  becomes  less 
1=1  " 


B. 


than  c^  + d^n. 

Stop  sampling  and  decide  in  favor  of  H, 
greater  than  c^^  + d^n. 


n 

as  soon  as  Ex.  becomes 
i=l  ^ 

n 


C.  Continue  sampling  as  long  as  + d^n  < Z ^i  ^1  ^ d^n. 

i=l 

Anderson's  calculations  of  the  ASN  for  his  modification  is  extremely 
complicated  and  involve  use  of  the  Wiener  stochastic  process.  However, 
his  ASN  for  intermediate  values  of  0,  between  6^  and  0^,  is  lower  than 
the  ASN  for  Read's  PSPRT.  Naturally,  due  to  its  optimum  property,  Wald's 
SPRT  has  a lower  ASN  at  6 = 9q  0 = 6^. 


Testing  Other  Than  Simple  Hypotheses 
The  SPRT  can  only  be  used  to  test  a simple  hypothesis  against  a 
simple  alternative  so  there  are  many  real  life  situations  in  which  it  is 
not  applicable.  For  example,  to  test  whether  a new  product  is  better  than 
an  already  existing  product  it  is  designed  to  replace  would  involve  test- 
ing Hq:0  = 0Q  vs.  Hj :0  > 0Q  and  the  SPRT  would  not  apply.  In  this  case, 
it  is  possible  to  modify  the  alternate  hypothesis  so  that  the  SPRT  could 
be  used.  Assuming  that  the  new  product  would  not  replace  the  old  one  un- 
less there  was  a significant  difference  in  some  performance  characteristic, 
the  alternate  hypothesis  can  be  written  as  H^:0  - 0^^  = d where  d is  the 
required  margin  of  difference  between  the  new  and  old  products.  Now  the 
alternate  hypothesis  is  in  simple  form  and  the  SPRT  may  be  applied. 

Another  special  case  is  when  the  decision  to  be  made  is  of  the  form: 
product  X is  Inferior  to  product  y or  there  is  no  difference  between  pro- 
ducts X and  y or  product  x is  superior  to  product  y.  This  is  equivalent 


32 
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to  testing  H ,:0  < 0 vs.  H :0  < 0 0 vs.  H,  : 0 > 0,  where  0 < 0.,  < 0., 

-i  i U/  3 ] 4 123 

< 0 . In  the  region  0 < 0 it  is  desirable  to  decide  that  product  X 
A 1 

is  inferior  to  product  y;  in  the  region  [02>  0^1  there  is  no  difference 
between  the  products;  and  in  the  region  0 > 0^  product  x is  superior  to 
product  y.  Sobel  and  Wald  [50]  and  Armitage  [6]  have  both  described 
methods  of  solving  this  problem. 

Sobel  and  Wald  define  their  three-decision  problem  as  deciding  be- 
tween hypotheses  H_j^:0  < a^^,  < 0 < a2  and  H^:0  > a^.  This  defini- 

tion leads  to  a division  of  the  parameter  space  into  five  regions.  Around 
a^  there  is  a region  (0^^,  02)  where  there  is  no  strong  preference  between 
H ^ and  but  where  it  is  strongly  desired  to  reject  Around  a2 

there  is  a region  (02*  where  there  is  no  strong  preference  between 

Hq  and  but  where  it  is  strongly  desired  to  reject  H For  0 ^ 0^^, 
the  desirable  decision  is  to  accept  H for  ^2  — ^ — ®3’  desirable 

decision  is  to  accept  and  for  0 ^ 0^,  the  desirable  decision  is  to 

accept  Given  this  formulation,  a wrong  decision  can  be  made  in  the 

following  manner: 

A.  Acceptance  of  or  for  0 02* 

B.  Acceptance  of  for  0^^  < 0 < 02. 

C.  Acceptance  of  or  for  02  £ ^ 

D.  Acceptance  of  H ^ for  0^  < 0 < 0^. 

E.  Acceptance  of  H , or  H„  for  0 > 0,. 

-10  — 4 

Sobel  and  Wald  then  consider  the  case  where: 

A.  Probablli*:y  of  a wrong  decision  < ® 

B.  Probability  of  a wrong  decision  ^ 0^^  < 0 < 0^. 

C.  Probability  of  a wrong  decision  ^ y^  for  0 0^. 


r 
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And  the  special  case  where  ~ Y2  ~ Y^  = Y for  all  values  of  6. 

Their  procedure  consists  of  conducting  two  concurrent  sequential 

probability  ratio  tests,  and  R2.  R^  is  used  to  test  the  hypotheses 

0 = G,  vs.  0=0.  and  R.  tests  0=6.  vs.  0=0,.  Their  decision  pro- 
1 2 2 3 A 

cedure  consists  of  the  following: 

A.  Both  Rj^  and  R^  are  computed  after  each  trial  until 

B.  Either:  one  ratio  leads  to  a decision  to  stop.  Then  this  ratio  is 
no  longer  computed  but  the'other  one  is  until  it  also  leads  to  a 
decision  to  stop. 

C.  Or:  both  ratios  lead  to  a decision  to  stop  at  the  same  stage. 

The  final  decision,  R,  can  be  made  from  the  results  shown  in  Table  8. 
Sobel  and  Wald  give  a proof  that  the  case  where  accepts  0^^  and  R^ 
accepts  6^  can  never  occur.  In  order  to  properly  define  the  SPRT's 
Rj^  and  R^,  it  is  necessary  to  either  be  given  values  A,  B,  A,  and  B 
which  form  the  boundaries  of  the  critical  regions  or  to  approximate  them 
based  on  the  upper  bounds  for  the  respective  probabilities  of  making  a 
wrong  decision.  These  approximations  are  shown  to  be: 


A = (1  ~ Y2)/Yj^ 

(2.16) 

B = Y2/(1  - Yj) 

(2.17) 

A = (1  ~ Y3)/Y2 

(2.18) 

B = Yj/d  - Y2) 

(2.19) 

The  special  case  Yj=Y2  ~ Y^  = Y is  easily  handled  by  substituting  in  the 
above  approximations. 

Armitage's  method  is  similar  except  that  It  involves  using  three 
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Table  8.  Decision  Process  for  Testing  Multiple  Hypotheses 


"l 

"2 

R 

If 

Accepts  0 

1 

and 

Accepts  0^ 

then 

Accepts 

»-l 

If 

Accepts  02 

and 

Accepts  0^ 

then 

Accepts 

»0 

If 

Accepts  02 

and 

Accepts  0^ 

then 

Accepts 

H. 

SPRT's  concurrently  so  that  all  possible  combinations  of  the  three  alter- 
native hypotheses  taken  two  at  a time  are  tested.  Armitage's  decision 
procedure  is  as  follows: 

A.  All  three  SPRT’s  are  examined  after  each  trial. 

B.  Sampling  is  continued  until  the  results  indicate  that  one  hypothesis 
is  preferred  over  both  of  the  other  two  hypotheses  based  on  the  SPRT's 
involving  those  hypotheses  and  the  preferred  one. 

C.  The  decision  is  then  made  in  favor  of  the  preferred  hypothesis. 

Armitage's  procedure  has  one  decided  advantage  over  that  of  Sobel 
and  Wald.  By  requiring  that  all  three  SPRT's  be  examined  until  completion, 
a definite  decision  in  favor  of  one  of  the  hypotheses  will  be  reached. 

Using  Sobel  and  Wald's  procedure,  it  is  possible  to  terminate  sampling 
before  a definite  decision  point  has  been  reached.  This  situation  is 
shown  in  Figure  2.  For  the  path,  OT,  shown  in  the  figure,  test  R is 
terminated  as  soon  as  line  AB  is  crossed  with  the  decision  being  made  in 
favor  of  Hp.  R^^  is  not  calculated  again  even  though  the  path  may  go  back 
across  the  line.  Test  R^  is  terminated  as  soon  as  line  AC  is  crossed  with, 
once  again,  the  decision  being  made  in  favor  of  H^.  Based  on  Table  8, 
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Che  terminal  decision  would  be  in  favor  of  but,  as  can  be  seen  from 
the  figure,  the  shaded  region,  Hq,  was  never  reached. 

Tests  on  Mean  of  a Normal  Distribution  with  Unknown  Variance 

Sequential  T-Test.  Once  again,  the  problem  is  to  test  the  hypo- 
thesis Bp  vs.  Hj^:0  ^ B-j^  > Bq-  Since  the  standard  deviation, 

a = will  always  be  a positive  number,  the  alternate  hypothesis  can 

be  expressed  as  H^^rO  ^ where  5 is  some  positive  number.  This 

can  be  rewritten  as  Hj^:B  - Bp  ^ 6a.  Since  the  value  of  O is  unknown,  the 
problem  is  to  find  a procedure  that  does  not  depend  on  0.  A random  vari- 
able defined  as  the  ratio  of  a N(0,1)  variable  to  the  square  root  of  a 
chi-square  random  variable  divided  by  its  degrees  of  freedom  follows  the 
t-d istributlon  and  the  formation  of  such  a random  variable  will  eliminate 
the  unknown  variance.  Govlndarojulu  [23]  has  formulated  a sequential  pro- 
cedure based  on  the  statistic: 


t = (n)  ^(x  - 0r,)/S 

n n 0 n 


(2.20) 


where  x = sample  mean  = ( Z x.)/n 

n i-1  i 


-.2  , , , , ,4 


S = sample  standard  deviation  = ( Z (x  - x)  /(n-1)) 
" i=l  ^ 


His  procedure  is  as  follows: 

A.  After  each  trial,  compute  t^  as  shown  in  (2.20). 


B. 

Continue  sampling 

as  long 

as 

B < 
n 

t 

n 

< A 

n* 

C. 

Stop 

sampling  and 

decide 

In 

favor 

of 

»0 

if  t 

n 

D. 

Stop 

sampling  and 

decide 

In 

favor 

of 

»1 

if  t 

n 

The  boundaries  A^  and  are  arbitrary  with  the  only  requirement 


being  that  they  be  chosen  so  that  the  procedure  terminates  finitely.  David 
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and  Kruskal  [17]  have  shovm  that  this  will  occur  when  A and  B are 

n n 

obtained  by  equating  the  ratio  of  a non-central  t distribution  to  a cen- 
tral t distribution  to  the  constants  A and  B from  Wald's  SPRT.  This  t- 
test  can  easily  be  modified  to  test  = 0^  vs.  Hj^:|0-9q1  ^ 6a  which 

is  equivalent  to  Hj^:0  4 0^  by  using  the  region  1 1 < A^  as  the 

region  in  which  sampling  is  continued. 

Two  Stage  T-Test.  Several  people  have  proposed  modifications  to 
the  t-test  that  require  an  initial  number,  n^,  of  observations. 
Govlndarajulu ' s [36]  Minimum  Probability  Ratio  Test  (MPRT)  is  equivalent 
to  Anderson's  modification  to  the  SPRT  only  for  the  case  where  the  vari- 
ance is  unknown.  For  the  case  where  a = 3,  Govlndarajulu  defines 


(n^  - 1) [ (2a) 


-2/(nQ-l) 


-1] 


(2.21) 


His  procedure  consists  of  the  following: 

2 

A.  Take  n-  initial  observations  and  compute  S as  defined  in  (2.20). 

0 n^ 

B.  For  each  observation  n 2.  after  observing  the  response  x^,  stop 
sampling  as  soon  as 


61  E (x.  - 6/2) |/S^  > C - n6^/4S^  , n > n„  (2.22) 

i=i  ^ "0“  "o  "o  - ° 

n 

C.  Decide  in  favor  of  if  Z (x , - 6/2)  < 0. 

^ i=i  ^ 

n 

D.  Decide  in  favor  of  H of  E (x  - 6/2)  > 0. 

^ i=l  ^ 

^ 2 2 2 

E.  Continue  sampling  if  6|  E (x  - 6/2) |/S  - n6  /4S 

1=1  ^ ”o  "o 


In  the  case  where  a 3,  the  procedure  may  still  be  used  by  replacing 
2a  with  a + 3 in  (2.21).  Govlndarajulu' s procedure  could  be  further 
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2 

modified  by  computing  C and  S after  each  observation. 

"o  "o 

Baker  (9)  as  modified  by  Hall  (24)  has  proposed  another  two  stage 
procedure  that  will  be  discussed  further  in  the  next  chapter. 
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CHAPTER  III 

DEVELOPMENT  OF  METHODOLOGY 


Introduction 

This  chapter  will  present  a formalized  procedure  for  building  a 
factorial  experiment  with  the  goal  of  reducing  the  required  sample  size 
without  reducing  the  amount  of  information  obtained  from  the  experimen- 
tation. This  information  may  include  data  about  the  significance  of  the 
various  factors,  the  levels  of  significance,  or  the  probabilities  of 
making  a type  I or  type  II  error.  The  method  developed  by  Russ  involves 
minimizing  the  expected  additional  system  cost  (EASC)  given  by 


EASC  = Cq  + 


N 

E C.  + a C + 6 C. 
1.1  - “ 6 


(3.1) 


where 

N 

C. 


C 

a 


a 


C 


6 


fixed  cost  of  testing.  This  can  be  considered  a fixed  cost 
Independent  of  the  design  structure, 
total  number  of  observations  taken. 

sampling  cost  of  i*"^  observation.  This  cost  will  be  assumed 

n 

be  a constant,  C , for  each  sample  taken  so  the  term  E C 
be  rewritten  as  NC  . 


to 

can 


s 

penalty  cost  for  a Type  I error.  Assumed  fixed  for  a given 
Operational  Test. 

probability  of  a Type  I error.  Set  at  an  acceptable  level  by 
the  test  designer. 

penalty  cost  for  a Type  II  error.  Assumed  fixed  for  a given 
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Operational  Test. 

B = probability  of  a Type  II  error.  Set  at  an  acceptable  level  by 
the  test  designer. 

For  a given  operational  test  all  of  the  costs  in  the  EASC  equation 
can  be  considered  as  fixed  so  the  equation  can  be  minimized  simply  by 
minimizing  the  number  of  observations  required  to  obtain  the  desired 
information, 

Basic  Assumptions 

Since  Operational  Testing  is  designed  to  test  one  proposed  system 
against  another  or  against  some  standard  for  comparison,  the  hypothesis 
to  be  tested  is  of  the  form: 


Hq:  ^ 

where  p = standard  for  comparison. 

Pq  = population  mean  for  the  item  being  tested. 

Operational  testing  will  require  that  the  prototype  item  exceed  the  stan- 
dard for  comparison  by  a certain  margin  before  the  decision  to  accept  the 
item  is  made  so  (3.2)  can  be  rewritten  as: 


Hq:  P - Pq  = 0 


Hr  P - Pq  = d 


(3.3) 


where  d is  the  required  margin  of  difference. 

Since  d will  be  some  positive  number,  it  can  be  expressed  as  some  constant 
multiple  of  the  population  standard  deviation,  also  a positive  number,  and 
(3.3)  can  be  rewritten  as: 
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Hq:  Vi  - Mg  - 0 

~ ^*0  “ 


(3.4) 


Given  the  nature  of  operational  testing,  the  assumption  is  also 

made  that  the  observed  responses  will  come  from  a normal  population  with 

2 2 

unknown  mean  u and  variance  O . The  value  of  o may  or  may  not  be  known 

2 

and  each  case  will  be  treated  separately.  If  O is  known,  the  problem 
is  greatly  simplified.  If  it  is  not  known,  a sequential  procedure  can 

2 

be  developed  but  an  alternate  solution  is  to  obtain  some  estimate  of  O 
as  soon  as  possible.  This  estimate  may  be  obtained  from  any  prior  test- 
ing done  on  the  system,  from  comparison  with  a similar  type  system,  or 
from  the  results  obtained  in  OT  I and  then  used  throughout  the  remaining 
operational  tests. 


Variance  Known 

If  the  variance  is  known,  the  sequential  probability  ratio  test 
can  be  employed  to  make  a decision  to  either  accept  the  null  or  alternate 
hypothesis  or  to  continue  sampling.  Since  (3.4)  is  in  the  form  of  a 
simple  null  hypothesis  against  a simple  alternate  hypothesis,  the  likeli- 


hood ratio  test: 


, 5:  F(x  |0  ) 

‘^In  i=l  ^ 


n F(x  |0  ) 
i=l  ^ 


(3.5) 


can  be  used  as  a basis  for  making  this  decision.  The  sequential  procedure 
should  include  a region  for  accepting  H^,  a region  for  accepting  and 
a region  in  which  the  decision  to  continue  sampling  is  made.  For  a normal 
population: 
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X -U 


I,  = 


n/2  exp(-  ~ i;  (x  - u)^)  (3.6) 

i=l  a (2110  ) 2o  i=l 


exp 


and  therefore 


1 12 

7r“2,n/2  7^  ^ 

In  _ (2110  ) 2o  1=1 

~ 2.n/2  7~2  ^ 

(2Tra  ) 2a  i=l 


(3.7) 


which  reduces  to 


"in 


on 


exp 


— = — ?,  X.  + :: 

O^  i=l  " 20^ 


(3.8) 


The  decision  rule  for  the  sequential  procedure  will  then  reduce  to: 


1)  Stop  sampling  and  accept  if  (3.8)  < B 

2)  Stop  sampling  and  accept  if  (3.8)  > A 


(3.9) 


';)  Continue  sampling  if  B < (3.8)  < A 

As  stated  in  Chapter  II,  values  of 


B . and  A - 

l-«  a 


(3.10) 


will  result  in  a sequential  test  with  the  ieslred  probabilities  of  error, 
a and  8. 

Substituting  (3.10)  into  (3.9),  taking  logarithms,  and  simplifying 
yields  the  following  sequential  procedure  for  a normal  population  with 
known  variance: 

0^  8 n(u^+ViQ) 

1)  Stop  sampling  and  accept  if  — ■ log  + — ^ J^x^ 
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g2  n(lij^+UQ) 

2)  Stop  sampling  and  accept  if  ^ ■_j"  log  + 2 ^ 


3)  Continue  sampling  if 


„2  „(„j+„d 


0^  , '-S  , 


where  is  as  hypothesized  in  (3.4)  and  + 60 


Variance  Unknown 

If  the  variance  of  the  normal  population  is  unknown,  the  problem 
is  considerably  more  difficult  since  the  likelihood  ratio  (3.8)  will 
depend  on  the  value  of  the  unknown  variance.  If  a reasonable  approxima- 
tion to  the  variance  can  be  obtained,  a sequential  probability  ratio  test 
can  be  performed  that  will  closely  approximate  the  test  for  the  variance 
known  case.  In  most  cases,  this  approximation  will  not  be  available.  Use 
can  then  be  made  of  the  fact  that  a random  variable  formed  as  the  ratio 
of  a N(0,1)  variable  to  the  square  root  of  a chi-square  variable  divided 
by  its  degrees  of  freedom  follows  a t distribution. 

Under  the  assumption  that  the  response  variable,  X^,  comes  from  a 

2 

N(u,o  ) population,  the  sum  of  n observed  responses  will  follow  a 
2 

N(u,no  ) distribution  and  the  statistic 

Z = (X  - u)/a/^  (3.11) 

will  follow  a N(0,1)  distribution.  A chi-square  random  variable  with  n 
degrees  of  freedom  is  the  sum  of  the  squares  of  n Independent  N(0,1) 


variables  so  the  quantity 


A4 


S ((X  - Vi)/o)^  (3.12) 

i=l  ^ 

will  be  distributed  as  chi-square  with  n degrees  of  freedom  and  the  t- 
statistic  proposed  by  Govindarajulu  in  (2.20)  can  be  formed. 

The  procedure  of  Baker  [9]  as  modified  by  Hall  [24]  incorporates 
the  required  margin  of  difference,  6,  in  the  formulation  of  the  test 
statistic 


r (S^  ) = (6  E (x.  - 6/2))/S^  (n  > n^)  (3.13) 

n "O  i=l  ^ "O  ~ ° 

in  the  proposed  two  stage  procedure.  Baker  develops  the  following  as 
appropriate  upper  and  lower  bounds  to  the  sampling  region: 
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C.  For  each  observation  n ^ n^,  after  observing  X^: 

2 

L.  Stop  sampling  and  decide  in  favor  of  H_  if  r (S  ) < b 

0 n no  Hq 

2.  Stop  sampling  and  decide  in  favor  of  H.  if  r (S  ) > a 

2 1 ^0  '^0 

3.  Continue  sampling  if  b < r (S  ) < a 

"o  " "o 

2 

The  modification  of  recomputing  S , a and  b after  each  obser- 

ve "o  '^0 

vation  may  be  easily  Incorporated  into  the  procedure.  This  test  would 

2 

then  be  equivalent  to  the  SPRT  (o  known)  with  wider  boundaries  to 

2 

account  for  the  fact  that  0 is  unknown  but  the  boundaries  would  converge 
to  the  SPRT  boundaries  as  n becomes  large. 

Solution  Procedure 

The  proposed  solution  algorithm  combines  the  sequential  procedures 
just  developed  with  a plan  for  systematically  building  a factorial  experi- 
ment in  order  to  obtain  as  much  information  as  possible  from  the  experi- 
ment in  as  few  observations  as  possible.  Use  is  made  of  screening 
experiments  in  building  the  full  factorial  in  order  to  eliminate  unneces- 
sary control  variables  from  future  experimentation. 

The  first  step,  once  the  requirement  to  conduct  operational  testing 
is  received,  is  to  determine  which  variables  can  be  expected  to  affect  the 
response  to  be  measured  and  how  much  they  can  be  controlled.  This  will 
set  the  size  of  the  full  experiment.  Once  the  number  of  observations 
that  can  be  developed  by  assuming  that  higher  order  interactions  are 
negligible  in  order  to  fractionate  the  full  factorial.  Once  the  first 
fraction  is  run,  a sequential  analysis  is  performed  to  determine  if  fur- 
ther experimentation  is  necessary.  If  it  is,  the  second  fraction,  with 
all  signs  reversed,  is  run  and  the  sequential  procedure  is  repeated.  In 
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6)  If  the  value  of  the  variance  is  unknown  or  If  there  is  no  reasonable 
estimate  for  the  variance,  go  to  8.  Otherwise  continue. 

7)  Perform  sequential  analysis  using  the  sequential  probability  ratio 
test  since  the  variance  is  known.  If  the  SPRT  results  in  a decision 
to  stop  sampling,  go  to  14;  otherwise,  go  to  9. 

8)  Perform  sequential  analysis  using  the  sequential  t test  since  the 
variance  is  unknown.  If  the  sequential  t test  results  in  a decision 
to  stop  sampling,  go  to  14;  otherwise,  continue. 

9)  Perform  the  next  block  of  experiments  with  all  signs  in  the  first 
block  reversed.  If  the  variance  is  known,  go  to  10.  If  the  variance 
is  not  known,  go  to  11. 

10)  Perform  sequential  analysis  using  the  SPRT.  If  the  decision  is  to 
stop  sampling,  go  to  14.  If  not,  go  to  12. 

U)  Perform  sequential  analysis  using  the  sequential  t test.  If  the 
decision  is  to  stop  sampling,  go  to  14.  If  not,  continue. 

12)  Perform  a screening  analysis  of  the  effects  due  to  the  main  factors 
in  order  to  determine  relative  significance  of  main  factors  and 
combinations  of  two-factor  interactions. 

13)  Determine  factors  and  interactions  of  interest  and  perform  the  next 
block  of  experiments  found  by  switching  the  signs  in  the  column  for 
the  factor  of  interest.  If  the  variance  is  known,  go  to  10.  If  the 
variance  is  not  known,  go  to  11. 

14)  Perform  an  ANOVA  to  determine  the  level  of  significance  of  all  factors 
of  Interest.  Eliminate  non-significant  factors  from  further  testing 
by  setting  them  at  some  common  level. 


15)  If  sequential  analysis  does  not  result  in  a decision  in  favor  of  one 
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hypothesis  or  another,  stop  sampling  and  perform  an  ANOVA  after  one 
observation  has  been  made  at  each  treatment  combination. 

The  proposed  methodology  will  be  illustrated  in  the  next  chapter 
by  means  of  an  example  based  on  an  actual  operational  test  and  the 

I sensitivity  of  the  input  parameters  demonstrated. 

1 

I 

i 

i 

I 

I 

I 

i 
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CHAPTER  IV 


DEMONSTRATION  OF  THE  APPROACH 

Introduction 

This  chapter  will  present  an  example  of  the  methodology  developed 
in  the  previous  chapter.  Hypothetical  results  from  an  Operational  Test 
are  presented  and  analyzed  in  the  classical  manner  currently  employed. 

Then  the  proposed  methodology  is  applied  to  the  same  results  with  a 
reduction  in  the  sample  size  required  to  gain  the  same  Information  from 
the  data.  Sensitivity  analysis  of  the  input  parameters  and  basic  assump- 
tions are  also  performed  to  demonstrate  the  robustness  of  the  procedure. 

Background 

The  Commander,  U.S.  Army  Operational  Test  and  Evaluation  Agency 
(OTEA)  has  been  given  the  requirement  to  conduct  Operational  testing  to 
evaluate  the  overall  effectiveness  of  the  new  Artillery  Locating  Radar, 
AN-TPQ-37,  designed  as  a replacement  for  the  system  currently  in  use. 

The  test  plan  calls  for  testing  several  different  performance 
aspects  of  the  Artillery  Locating  Radar  (ALR) , one  of  which  is  its  ability 
to  detect  hostile  artillery  fire.  Since  the  radar  cannot  locate  hostile 
artillery  unless  it  first  detects  it,  the  most  critical  issue  for  this 
test  is  the  percentage  of  hostile  artillery  rounds  detected.  The  manu- 
facturer has  determined  that  five  factors  should  Influence  the  performance 
of  the  radar.  They  are  the  threat  array  employed  hy  tlie  enemy,  the  use 
of  electronic  counter  measures  (ECM)  by  the  enemy,  the  rate  of  hostile 
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fire,  the  range  from  the  ALR  to  the  enemy  threat,  and  the  sector  the  ALR 
is  searchl*'  All  of  these  factors  can  be  set  at  two  levels  except  for 
the  threat  array.  That  can  be  set  at  one  of  four  levels  to  represent 
the  typical  composition  of  enemy  artillery  units  that  the  ALR  is  likely 
to  be  deployed  against.  Since  four  is  a multiple  of  two,  the  threat 
array  factor  can  be  represented  as  two  pseudo-factors  each  with  two  levels 
and  the  entire  experiment  can  be  represented  as  a 2^  factorial  experiment. 
The  six  factors  and  the  levels  of  each  factor  are  shown  in  Table  9. 


Table  9. 

Factors  in  2^  Factorial 

Experiments 

Factor 

Low  Level 

High  Level 

A - ECM 

Not  Employed 

Employed 

B - Rate  of  Fire 

Slow 

Fast 

C - Range 

Short  < 10,000  KM 

Long  > 10,000  KM 

D - Sector 

Narrow  (±  15°  of  center 

Wide  (15-45°  of  center) 

E - Threat  Array 

I 

II 

F - Threat  Array 

III 

IV 

Where  THREAT  ARRAY 

1=1  enemy  battery 

II  = 2 enemy  batteries 

III  = 1 enemy  battalion 

IV  = 2 enemy  battalions 


The  Commander,  OTEA,  has  stated  that  the  ALR  be  tested  by  deter- 
mining the  percentage  of  hostile  artillery  round  detected  from  firings 
taken  at  each  of  the  64  possible  treatment  combinations.  The  analysis 
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method  used  will  be  Analysis  of  Variance  (ANOVA)  with  all  third  order 

Tnd  higher  interactions  assumed  to  be  negligible  and  pooled  to  form  an 

estimate  of  the  error.  The  standard  for  comparison  is  the  current 

system  which  is  detecting  50%  of  all  hostile  artillery  rounds.  The 

measure  of  effectiveness  employed  will  be  the  percentage  of  rounds 

located.  The  principle  purpose  of  the  test  will  be  to  determine  whether 

or  not  the  mean  percentage  of  rounds  detected  by  the  ALR  exceeds  the 

current  system  by  some  multiple  of  the  standard  deviation.  It  is  assumed 

that  the  percentage  of  rounds  detected  comes  from  a normal  distribution 

2 

with  unknown  mean,  u,  and  variance,  a . Therefore,  the  testing  will 
consist  of  a test  of  vs.  H^:u  “ 1*0  "*■  where  6 is  some 

positive  constant. 

Classical  Methods 

OTEA  conducted  test  firings  at  all  64  treatment  combinations  and 

determined  the  percentage  of  rounds  detected  from  each  firing.  Time  limi- 

6—3 

tations  precluded  any  more  than  eight  test  firings  per  day  so  a 2 
resolution  III  design  with  I = ABD  = ACE  = BCE  = BCDE  = ACDF  = ABEF  = DEF 
as  the  generating  relationship  was  used  to  fractionate  the  design  and 
then  the  eight  different  blocks  were  fired  in  random  order.  The  results, 
percentage  of  rounds  detected,  and  the  ANOVA  for  the  entire  experiment 
are  shown  in  the  appendices.  The  results  of  the  ANOVA  indicate  that  only 
the  factors  relating  to  ECM,  rate  of  fire,  and  sector  of  search  are  sig- 
nificant so  OT  II  should  be  conducted  using  only  those  factors. 

Proposed  Methodology 

Using  the  same  data  and  restrictions  as  in  the  actual  OT  I,  the 
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following  example  will  Illustrate  the  proposed  sequential  procedure. 

The  first  eight  observations  are  obtained  from  the  block  gene- 
rated by  I = ABD  = ACE  = BCF  = BCDE  = ACDF  = ABEF  = DEF  and  consists  of 
treatment  combinations  def  (37),  af  (36)  be  (40),  abd  (87),  cd  (61), 
ace  (34),  bcf  (46)  and  abcdef  (89)  where  the  values  in  parentheses 
represent  the  observed  response  for  that  treatment  combination.  Based 
on  these  observed  responses,  6 = 1,  a = .10,  and  B = .10,  the  following 
computations  are  performed: 


y = 430/8  = 53.75 

(4.1) 

Sg  = 3635.5/7  = 519.35 

(4.2) 

Ag  = '5(7)(.r^^^-l)  = 3.26 

(4.3) 

Bg  = -b(7)(.r^^^-l)  = -3.26 

(4.4) 

rg(Sg)  = 6(E(x^  - 6/2))/Sg  = 426/519.35  = .82 

(4.5) 

Since  B_  < ro(So)  < A„,  the  decision  is  made  to  continue  sampling. 

O O o o 

6“3 

The  alias  structure  for  this  2 resolution  III  design  is  shown 
in  Table  10.  Assuming  that  all  third  order  and  higher  interactions  are 
negligible,  each  major  factor  is  aliased  with  two  two-factor  interactions. 
In  order  to  separate  the  main  effects  from  their  aliased  two-factor  inter- 
actions, the  next  block  run  will  be  the  same  as  the  first  but  with  all 
signs  reversed.  The  defining  relationship  for  this  block  will  then  be 
I = -ABD  = -ACE  = -BCF  = BCDE  = ACDF  = ABEF  = -DEF.  The  block  consists 


of  the  following  treatment  combinations  and  their  observed  responses: 
ahc  (69),  bcde  (56),  acdf  (67),  cef  (10),  abef  (69),  bdf  (56),  adc  (75), 
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6*“  3 

Table  10.  Alias  Structure  for  2^^^^  Design 


Factor  A L 1 A S E S 


! 

ABD 

ACE 

BCF 

BCDE 

ACDF 

ABEF 

DEF 

1 ^ 

BD 

CE 

ABCF 

ABCDE 

CDF 

BEF 

ADEF 

: B 

AD 

ABCE 

CF 

CDE 

ABCDF 

AEF 

BDEF 

i C 

ABCD 

AE 

BF 

BDE 

ADF 

ABCEF 

CDEF 

D 

AB 

ACDE 

BCDF 

BCE 

ACF 

ABDEF 

EF 

E 

ABDE 

AC 

BCEF 

BCD 

ACDEF 

ABF 

DF 

1 

i F 

ABDF 

ACEF 

BC 

BCDEF 

ACD 

ABE 

DE 

1 

. af 

BDF 

CEF 

ABC 

ABCDEF 

CD 

BE 

ADE 

and 

(l)(2-i). 

Sequent ia 1 

C H 1 v iJ 

lat  ions  resul t 

in  the 

following: 

V • 856/16  » 53.5 

(4.6) 

S^^  - 502.4 

1 o 

(4.7) 

,,  - 2.695  , B,^  » -2.695 

J n in 

(4.8) 

(S^^)  - 848/502.4  = 1.69 

1 n 

(4.9) 

Since  R,.  < r,,(S,.)  < A,,,  the  decision  is  made  to  continue  sampling. 

1 n in  1 b lo 

Before  the  next  block  is  run,  however,  an  analysis  of  the  results 
obtained  so  far  is  performed.  The  results  are  shown  in  the  appendices. 
Based  on  the  relative  magnitude  of  the  effects,  it  would  appear  that 
factors  A,  B,  and  D were  significant  while  factors  C,  E.  and  F were  not. 
Since  the  largest  interaction  term  is  due  to  BD  and  CE,  It  is  also 
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I. 

!: 


I , 
I 
( 

!• 


possible  that  the  BD  interaction  is  significant.  Since  the  decision  has 
already  been  made  to  continue  sampling,  the  next  block  run  will  be  the 
same  as  the  first  block  except  that  all  of  the  signs  in  the  column  for 
factor  B will  be  reversed.  This  will  isolate  factor  B and  all  of  its  two 
factor  interactions  and  will  provide  a better  estimate  of  whether  or  not 
the  BD  interaction  is  significant. 

The  third  block  then  consists  of  the  following  treatment  combina- 
tions and  their  observed  responses:  bdef  (67),  abf  (74),  e (’■'’',  ad  (66), 
bed  (67),  abce  (72),  cf  (11),  aedef  (75).  The  sequential  calculations 
result  in  the  following: 


y = 1298/24  = 54.08 

(4.10) 

S^,  = 563.99 

24 

(4.11) 

24  = 2.55  B^^  = -2.55 

(4.12) 

(S^  ) = 1286/563.99  = 2.28 

24 

(4.13) 

2 

Since  B„,  < r„,(S„,)  < A„,,  the  decision  is  made  to  continue  sampling. 

24  24  24  24 

An  analysis  of  the  results  of  the  data  from  the  third  block  is  shown  in 
the  appendices.  Once  again,  factors  A,  B,  and  D appear  to  be  significant 
while  the  BD  interaction  is  still  in  doubt.  After  the  second  block,  the 
effect  in  question  was  due  to  the  BD  and  CE  Interactions  so  it  is  possible 
that  this  was  due  to  the  CE  rather  than  the  BD  interaction.  Since  the 
decision  has  been  made  to  continue  sampling,  the  next  block  run  will 
change  the  signs  in  the  column  for  factor  c to  isolate  factor  c and  Its 
two  factor  interactions.  This  block  contains  the  following  treatment 
combinations  and  observed  responses:  edef  (48),  acf  (41),  bee  (30), 


I 

I 

J 


1 
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abed  (95),  d (57),  ae  (46),  bf  (31),  and  abdef  (86).  The  sequential 
calculations  result  in  the  following: 


y = 1732/32  = 54.125 
= 550.31 

= 2.48  , 

’’32^^^^  = 1716/550.31  = 3.  12 


(4.14) 

(4.15) 

(4.16) 

(4.17) 


Since  ’‘32^^32^  ^ ^32'  decision  is  made  to  stop  sampling  and  accept 

= Uq  + (So  = 50  + o.  This  means  that  the  ALR  has  performed  better 
to  date  than  the  current  system.  The  analysis  of  the  data  from  the 
fourth  block  is  shown  in  the  appendices.  Once  again  factors  A,  B,  and 
D appear  to  be  significant  while  the  CE  interaction  does  not.  Since  the 
decision  has  been  made  to  stop  sampling,  the  recommendation  made  at  this 
point  would  be  to  perform  OT  II  using  only  the  factors  for  ECM,  rate  of 
fire,  and  sector.  The  other  factors  would  be  set  at  some  acceptable 
standard  level  and  left  there. 

An  interesting  sidelight  Illustrated  by  this  example  is  the 
process  of  collapsing  a design  in  k variables  to  a smaller  design  in 
p < k variables  by  eliminating  non-significant  variables  from 
consideration.  To  Illustrate  this  procedure,  the  experimenter  was 
relatively  sure  that  the  variables  C,  E,  and  F were  not  significant  after 
the  second  block  of  eight  experiments  had  been  performed  and  the  screening 
analysis  conducted.  Had  the  decision  been  made  to  eliminate  those  vari- 
ables from  further  consideration  at  that  point  by  setting  them  at  some 
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standard  level,  the  next  blocks  of  eight  experiments  performed  would 

3 

consist  of  a replicated  1/2  fraction  of  a 2 factorial  in  the  signifi- 
cant variables  A,  B,  and  D.  This  is  shown  in  the  table  of  plus  and 
minus  signs  in  Table  11.  In  addition,  the  two  blocks  combined  form  a 
replicated  full  factorial  in  the  three  factors  A,  B,  and  D. 


Table  11.  Collapsed  Design  in  Three  Variables 


BLOCK  3 

BLOCK 

A 

Treatment 

Combination 

A 

B 

Treatment 

D Combination 

A 

D 

bdef 

- 

+ 

+ edef 

- 

- 

+ 

abf 

+ 

+ 

_ acf 

+ 

- 

- 

e 

- 

- 

bee 

- 

+ 

- 

ad 

+ 

- 

+ abed 

+ 

+ 

+ 

bed 

- 

+ 

+ d 

- 

- 

+ 

abce 

+ 

+ 

ae 

+ 

- 

- 

cf 

- 

- 

bf 

- 

+ 

- 

aedef 

+ 

- 

+ a bdef 

+ 

+ 

+ 

Sensitivity 

This 

section 

will  demonstrate  how  the  conclusions 

reached 

in  the 

sequential 

analysis 

would 

vary  should  any  of  the 

input 

parameters 

be 

changed . 

ct  and  6 

Table  12  shows  the 

different  decisions  made  if 

the 

probabillt ies 

of  Type  I (a)  and  Type  II  (S)  errors  are  changed.  These  results  would 
appear  to  be  intuitively  correct.  As  the  values  for  n and  p are  increased 
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the  boundaries  for  making  a decision  come  closer  together  since  the 
experimenter  is  more  willing  to  make  an  error  while  for  smaller  values 
of  ni  and  3,  the  boundaries  are  farther  apart  because  the  experimenter  is 
willing  to  make  an  error. 

Table  12.  Sensitivity  of  a and  6 Errors 


01,6 

n = no.  of  Observations 

N 

n n 

Decision 

.05 

8 

4.74 

-4.74 

.82 

Continue 

16 

3.68 

-3.68 

1.69 

Continue 

24 

3.42 

-3.42 

2.28 

Continue 

32 

3.30 

-3.30 

3.12 

Continue 

.15 

8 

2.52 

-2.52 

.82 

Continue 

16 

2.16 

-2.16 

1.69 

Cont inue 

24 

2.06 

-2.06 

2.28 

Stop-Accept  1 

32 

2.02 

-2.02 

3.12 

N/A 

.20 

8 

2.04 

-2.04 

.82 

Continue 

16 

1.80 

-1.80 

1.69 

Continue 

24 

1.73 

-1.73 

2.28 

Stop-Accept  1 

32 

1.70 

-1.70 

3.12 

N/A 

Sequential  Parameters 

The  sequential  procedure  employed  involves  recomputing  A , B , 

2 

and  S after  each  block  of  observations.  Table  13  shows  the  change 

"o 

in  the  results  if  this  modification  is  not  employed  by  using  the  values 


Aq  = 3.26,  B„  = -3.26  and  S_  = 519.35  throughout  the  sequential  analysis 

o o o 
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Table  13.  Sensitivity  of  Input  Parameters 


s2 

3 

00 

r (S  ) 
n n 

n 

n 

Decision 

Decision 

8 

519.35 

.82 

Continue 

.82 

Continue 

16 

502.4 

1.63 

Continue 

1.69 

Cont inue 

24 

563.99 

2.48 

Continue 

2.28 

Continue 

32 

550.31 

3.30 

Stop-Accept 

3.12 

Continue 

or  is  only  partially  employed  by  using  the  values  for  A_  and  B_  through- 

o o 

2 

out  but  recomputing  S after  each  block.  Eliminating  the  convergence  of 

n 

the  boundaries  as  more  samples  are  taken  results  in  the  decision  to  con- 
tinue sampling  after  32  samples  have  been  taken  but  would  result  in  the 
decision  to  stop  sampling  and  accept  after  the  next  block  of  eight 
for  a total  of  40  samples. 

Improvement  Required  (&) 

This  example  was  run  with  the  new  system  required  to  outperform  the 
old  system  by  a factor  of  one  standard  deviation.  This  value  was  arbi- 
trarily selected  by  the  test  designer.  Table  14  shows  the  changes  in 
the  decision  making  procedure  as  a result  of  changing  the  value  of  6. 

Variance  Known 

The  preceding  example  Illustrated  the  sequential  analysis  procedure 
when  the  observations  were  assumed  to  have  come  from  a normal  population 
with  unknown  mean  and  variance.  In  many  real  life  cases,  there  may  be 
some  prior  data  available  so  that  a good  estimate  for  the  variance  may 
be  obtained.  Assuming  that  the  previous  testing  had  been  conducted,  the 
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Table  14.  Sensitivity  of  (5 


n 

6 

r (S^) 
n n 

Decision 

8 

. 5 

.41 

Continue 

1.5 

1.22 

Continue 

2.0 

1.63 

Continue 

16 

.5 

.85 

Continue 

1.5 

2.52 

Continue 

2.0 

3.34 

Stop-Accept 

»1 

24 

.5 

1.14 

Continue 

1.5 

3.40 

Stop-Accept 

»1 

32 

.5 

1.57 

Continue 

Error 

= 52.35  could 

be  used  as  an 

estimate  of  the 

variance  for  future 

testing. 

The  problem 

then  resolves 

to  testing  Hq:u 

= 50  vs.  H^:u  =50 

+ la  ~ 57,2  and  the  Sequential  Probability  Ratio  Test  can  be  employed. 
Using  a = P = .10,  the  boundaries  for  the  SPRT  become: 

(4.18) 

A = = ^ = 9 (4.19) 

« . 1 


As  shown  in  Chapter  III,  the  SPRT  consists  of  the  following: 
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3)  After  24  observations: 


T.  y . = 1298 
1 


(4.26) 


B.,,  = (-16.05)  + (53.6)(24)  = 1270.35 
24 


A.,  = 15.98  + (53.6)(24)  = 1302.38 
24 


(4.27) 


(4.28) 


Since  ^ ' ^24’  decision  to  continue  sampling  is  made. 

4)  After  32  observations: 


E y.  = 1732 
1 

B^2  = -16.05  + (53.6)(32)  = 1699.15 


(4.29) 


(4.30) 


B^2  = 15.98  + (53.6)(32)  = 1731.18 


(4.31) 


Since  ^ ^ ^32’  decision  to  stop  sampling  and  accept  is 

made.  The  Sequential  Probability  Ratio  Test  resulted  in  the  same  deci- 
sion after  the  same  number  of  observations  as  the  sequential  t-test. 

An  explanation  for  the  fact  that  the  test  did  not  terminate  sooner  when 
the  variance  was  assumed  known  can  be  found  in  the  fact  that  the  mean 

of  the  64  observations  used  was  53.25  which  is  close  to  half-way  between 

the  two  hypothetical  values  of  50  and  57.2  and  the  SPRT  performs  best  at 

values  near  the  hypothesized  values  and  worse  at  values  close  to  the 

mid-point  of  the  hypothesized  values.  In  spite  of  this  fact,  the  SPRT 
still  terminated  in  half  the  number  of  observations  required  by  the 
classical  methods  currently  employed. 
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CHAPTER  V 


RELATED  APPLICATIONS 

Introduction 

The  proposed  methodology  developed  in  Chapter  III  was  applied  to 
a specific  situation  in  Chapter  IV.  The  situation  described  in  Chapter 
IV,  although  specific  in  nature,  was  really  taken  from  a general  class 
of  problems,  2'^  factorials,  to  which  the  methodology  may  be  applied. 

This  chapter  will  present  several  highly  specific  procedures  which  may 
be  combined  with  the  proposed  methodology  in  certain  situations  to  gain 
even  more  benefits  from  the  procedure. 

Major  and  Minor  Variables 

In  many  situations,  prior  knowledge  of  the  system  or  of  a similar 
system  may  allow  the  experimenter  to  determine  which  factors  will  defi- 
nitely make  a significant  contribution  prior  to  the  start  of  experimen- 
tation. There  may  be  seveial  other  factors  about  whose  contribution  the 
experimenter  is  unsure  and,  therefore  desires  further  information.  By 
classifying  the  £ priori  significant  variables  as  major  variables  and 
the  remainder  as  minor  variables,  use  can  be  made  of  the  properties  of 
blocking  a factorial  experiment  to  gain  more  information  from  the 
experiment.  Generally,  the  experimenter  will  desire  an  estimate  of  the 
main  effect  and  all  interactions  for  the  major  variables  but  he  will  be 
willing  to  assume  that  all  interactions  involving  minor  variables  are 
negligible  so  will  only  want  an  estimate  of  the  main  effect  for  the  minor 


variable.  The  procedure  involving  major  and  minor  variables  is  specific 
in  that  it  requires  a resolution  IV  design  blocked  into  a number  of 
blocks  equal  to  the  number  of  major  variables.  By  associating  each 
major  variable  with  a block  and  the  minor  variables  with  the  experi- 
mental variables,  significance  of  a block  effect  will  indicate  signifi- 
cance of  a major  variable  and  an  effect  due  to  an  interaction  between 
blocks  will  indicate  an  interaction  between  major  variables.  It  may  be 
that  an  estimate  for  an  effect  will  involve  a combination  of  a major 
variable  and  some  number  of  minor  variable  interactions  but  this  situation 
poses  no  problem  since  the  experimenter  has  assumed  that  all  order  inter- 
actions Involving  minor  variables  are  negligible.  A specific  example  of 

8-4 

the  use  of  major  and  minor  variables  is  the  2 resolution  IV  design 

which  can  be  used  to  investigate  eight  minor  and  three  major  variables. 

Since  the  design  contains  16  points,  the  first  step  in  constructing  the 

4 

design  matrix  is  to  write  down  a full  2 factorial  design  in  four  of  the 
minor  variables.  The  remaining  four  minor  variables  are  then  expressed 
as  three  factor  interactions  of  the  first  four  variables  and  the  major 
or  blocking  variables  are  expressed  as  two  factor  interactions  of  some 
pair  of  minor  variables.  The  design  matrix  for  this  design  is  shown  in 
Table  15.  The  design  is  then  separated  into  eight  blocks  of  two  runs 
each  by  combining  the  treatment  combinations  that  have  the  same  signs  on 
Bj,  B^,  and  B^,  i.e.,  the  sets  (+,+,-), 

(-,-,+),  (+,-,+),  (-,+,+),  and  (+,+,+),  to  form  the  eight  blocks.  One 

interesting  thing  to  note  when  the  treatment  combinations  are  paired  in 
this  manner  is  that  the  two  treatment  combinations  in  the  same  block 


r 


have  opposite  signs  for  every  factor. 


Table  15.  Design  Matrix  for  Major  and  Minor  Variables 


Minor  Variables Major  Variables 

A B C D E=ABC  F=ABD  G=ACD  H=BCD  B^=AB  B^=AC  Bj=AD 

_____  _ _ _ + + + 

+ --  -+  + + - - - - 

- + + + - + - + + 

+ + --  - - + + + - - 

- + + + - + 

+ _ + __  + _ + _ + _ 

- + + + + - - - + 

+ + + -+  - - - + + - 

---+  - + + + + + - 
+ --  ++  - - + - - + 

- + -+  + - + - - + - 

+ + -+  - + - - + - + 

--  + ++  + - - + - - 
+ - + +-  - + - - + + 

- + + +-  - - + - - - 

+ + + ++  + + + + + + 

Sequential  Facto rial  Es t Imatlon 

In  certain  cases,  the  experimenter  may  desire  further  information 
about  the  model  that  represents  the  system  under  investigation.  For  an 
experiment  involving  P variables  and  assuming  that  all  third  order  and 
higher  interactions  are  negligible,  the  general  model  can  be  written  as: 


F 
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E(y)  = Br,  + S B.  X.  + ?;  Y,  B..  x.  x. 
i=l  i>  i = l 


(5.1) 


Least  squares  estimates  of  all  of  the  coefficients,  B's,  in  the  model 
can  be  obtained  from 


B = (x'x)  ^x'y 


(5.2) 


where  B = 


X = 


1 D 


12 


p-1  p 


D = design  matrix  of  experiment  and  y = column  vector  of  observations 
as  long  as  the  number  of  observations  is  greater  than  or  equal  to  the 
number  of  coefficients  for  which  an  estimate  is  desired.  On  the  high 
speed  computers  available  today,  equation  (5.2)  can  be  evaluated  quickly 
and  easily. 

Hunter  [31]  has  developed  a similar  method  where  the  computations 
may  be  made  easily  on  a hand  calculator  in  case  the  experimenter  does 
not  have  ready  access  to  a computer.  His  method  requires  that: 

1)  The  model  contain  no  more  than  q < N coefficients  and  an  experimental 
design  containing  N experiments  has  been  completed. 

2)  The  estimates  provided  by  prior  blocks  must  be  mutuallv  orthogonal 

‘’2 

witli  variance  equal  to  — . 

^ mN 
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3)  The  added  row  vectors  must  be  row-wise  orthogonal  ^ iVj). 

Conditions  2 and  3 above  are  satisfied  by  a 2 factorial  design  if 
q = N.  Once  an  initial  block  of  N runs  has  been  obtained,  initial  esti- 
mates of  the  coefficients  can  be  obtained  from  equation  (5.2).  From  then 
on.  Hunter's  Predictor-Corrector  equation  can  be  used  to  update  the  esti- 
mates of  the  coefficients  after  each  run.  The  P-C  equation  is  given  by: 


B = B + .5^,  (y  .-y . )r . 

mN+q  1= 1 1 1 1 


(5.3) 


where:  B = (q  ^ 1)  vector  of  revised  estimates. 

B = (q  < 1)  vector  of  estimates  provided  by  prior  block(s). 

N = number  of  runs  in  a block. 

n = number  of  runs  completed  in  current  block. 

q = number  of  coefficients  in  the  model. 

m = number  of  blocks  of  N runs  completed. 

T 

r^  = (1  q)  row  vector  in  matrix  of  independent  variables 

associated  with  i*”^  experiment  1 = l,2,...,n  ^ N. 

T 

V.  = new  observation  associated  with  r.. 

1 1 

y^  = r^  B = predicted  response  for  i*"^  experiment. 

Once  the  initial  estimates,  B,  have  been  obtained,  they  are  used 

* 

throughout  the  next  block.  The  revised  estimates  B are  computed  after 
each  experimental  run  using  the  B computed  after  the  previous  block.  The 
estimates,  B,  are  updated  after  each  complete  block  is  finished  and  the 
updated  estimates  are  then  used  throughout  the  entire  next  block.  The 
variance  of  each  revised  estimate  can  be  obtained  from: 


V(b*)  = [1  - 

mN  mN+q 


(5.4) 
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wltere  a Is  the  population  variance  and  m,  N,  n,  q are  as  defined 
previously.  The  procedure  requires  computing  the  Inverse  of  a matrix 

only  after  the  initial  block  of  N observations  have  been  made.  i 

i 

This  procedure  is  particularly  useful  when,  for  some  reason,  it  ' 

becomes  impossible  to  complete  the  experimentation.  The  change  (increase)  1 

in  the  sum  of  squares  due  to  error  (deviation)  for  each  experimental  run 
can  be  computed  from 


ASSD 


mN  . 
mN+q  ^^i 


(5.5) 


so  that  the  analysis  of  variance  table  can  be  updated  at  the  completion 
of  each  run.  This  provides  the  experimenter  with  a valid  ANOVA  table  in 
the  event  that  a complete  block  of  N experiments  cannot  be  completed. 

The  complete  ANOVA  after  m blocks  of  N experiments  is: 


Source 

!§. 

mN 

9 

SSY  = Crude  SS 

I. 

1=1 

L 

^i 

mN 

mN 

SSR  = Regression  SS 

X 

i=l 

^i 

q 

mN 

9 

SSI)  = Error  SS 

y. 

i=l 

(Yi  - y,) 

mN- 

To  satisfy  the  requirement  that  q = N it  may  be  necessary  to 
Introduce  some  slack  variables.  The  easiest  way  to  do  tliis  is  to  pick 
some  higher  order  interact  ion (s)  that  may  be  of  interest  and  include 
them  in  the  regression  model  so  that  an  estimate  of  their  coefficients 


will  also  be  obtained.  An  example  applying  this  procedure  to  the  data 
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from  the  example  in  Chapter  IV  is  shown  in  the  appendices. 

Blocking  Fractional  Factorials 

In  the  example  illustrated  in  Chapter  IV,  each  set  of  eight  experi- 
ments consisted  of  a 1/8  fraction  of  the  full  2^  factorial.  After  two 
' sets  of  eight  experiments  had  been  performed,  the  experimenter  had,  in 

effect,  performed  a 1/4  fraction  of  the  2^  in  two  blocks  and  the  experi- 
I ment  could  be  analyzed  as  such  at  this  point. 

I To  illustrate  this  procedure,  consider  the  first  two  sets  of  eight 

experiments  performed  in  the  example  in  Chapter  IV.  They  now  form  a 1/4 
6 6“  2 

fraction  of  the  2 or  a 2 with  generating  relation  I = BCDE  = ACDF  = 

ABEF.  The  alias  structure  for  this  design  is  shown  in  Table  16.  Since 
there  are  two  alias  sets  containing  only  three  factor  interactions,  one 
of  these  is  confounded  with  the  two  blocks  run.  In  this  case,  the  inter- 
action ABD  and  its  aliases  were  confounded  with  blocks.  The  eight  treat- 
i ment  combinations  in  the  first  block  performed  all  contain  an  odd  number 

of  letters  in  common  with  ABD  and  the  eight  treatment  combinations  in 
the  second  block  performed  all  contain  an  even  number  of  letters  in  com- 
mon with  ABD.  The  experimenter,  in  performing  an  Analysis  of  Variance  at 
this  point  could  then  extract  one  degree  of  freedom  for  blocks.  An 
.analysis  of  the  effect  due  to  blocking  could  provide  the  experimenter 

with  some  idea  of  a training  or  learning  process  or  an  effect  such  as  ! 

I 

1 


L 


weather  that  may  be  having  .an  effect  from  block  to  block. 
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Table  16. 

Alias  Structure 

fi  9 

for  2 ~ with  I = BCDE  = ACDF  = 

ABEF 

I 

BCDE 

ACDF 

ABEF 

A 

ABCDE 

CDF 

BEF 

B 

CDE 

ABCDF 

AEF 

C 

BDE 

ADF 

ABCEF 

D 

BCE 

ACF 

ABDEF 

E 

BCD 

ACDEF 

ABF 

F 

BCDEF 

ACD 

ABE 

AB 

ACDE 

BCDF 

EF 

AC 

ABDE 

DF 

BCEF 

AD 

ABCE 

CF 

BDEF 

AE 

ABCD 

CDEF 

BF 

AF 

ABCDEF 

CD 

BE 

BC 

DE 

ABDF 

ACEF 

BD 

CE 

ABCF 

ADEF 

ABC 

ADE 

BDF 

CEF 

ABD 

ACE 

BCF 

DEF 

■ -- - A 
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CHAPTER  VI 

CONCLUSIONS  AND  RECOMMENDATIONS 

Limitations  of  the  Research 

This  research  is  limited  in  application  to  univariate  response 

models.  It  is  assumed  that  the  response  comes  from  a normal  population 

2 

with  unknown  mean,  p,  and  variance,  O , which  may  or  may  not  be  known. 

The  approach  is  demonstrated  only  for  a 2^^  factorial  experiment  that  is 
fractionated  into  a Resolution  III  design  but  is  easily  extended  into 
a factorial  experiment  where  the  factors  take  on  any  number  of  levels 
as  long  as  the  full  factorial  can  be  fractionated  into  a Resolution  III 
or  higher  design. 

Conclusions 

This  research  accomplished  three  objectives: 

A.  An  approach  to  systematically  building  a factorial  experiment  through 
the  use  of  screening  experiments  was  demonstrated.  This  allows  tae 
experimenter  to  obtain  as  much  information  as  possible  from  a fixed 
set  of  resources. 

B.  A method  of  sequentially  analyzing  the  data  from  a fractionated 
factorial  experiment  was  demonstrated.  This  allows  the  experimenter 

to  obtain  a fixed  amount  of  information  from  a reduced  set  of  resources. 

C.  The  proposed  methodology  combined  the  above  two  methods  to  systemati- 
cally build  a factorial  experiment  while  conducting  a sequential 
analysis  of  the  data  at  the  end  of  each  block  of  the  factorial 


experiment.  This  allows  the  experimenter  to  gain  the  maximum  amount 
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of  information  from  a minimum  amount  of  resources. 

Recommendations 

Although  this  research  considers  only  a univariate  response  model, 
operational  testing  often  involves  the  testing  of  several  Measures  of 
Effectiveness  (MOE).  Therefore,  it  is  recommended  that  future  research 
in  this  area  be  directed  at  the  development  of  a methodology  to  handle 
the  case  of  multiple  response  models. 

This  research  also  was  demonstrated  only  for  a 2*^  factorial 
experiment.  Future  work  in  the  area  should  be  directed  at  applying  the 
methodology  to  experiments  in  which  the  factors  appear  at  other  than  two 
levels  or  to  other  than  factorial  experimental  designs. 

As  the  results  of  more  operational  tests  become  available,  it  is 
recommended  that  the  U.S.  Army  Operational  Test  and  Evaluation  Agency 
apply  the  proposed  methodology  to  the  completed  test  data  as  a further 
test  of  its  validity  and  worth  as  a viable  analysis  method  for  their 

eventual  adoption. 

I 

I 

i 
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APPENDIX  A 

DATA  FROM  THE  FULL  2^  FACTORIAL  EXPERIMENT  AND 
ANALYSIS  OF  VARIANCE  TABLE 


This  appendix  contains  the  data  from  the  full  2 factorial  experi- 


ment 

and  an 

Analysis  of  Variance 

Table  for 

the 

entire  data 

set . 

Data  from  the 

Full  2^ 

Factorial  Experiment 

(1) 

2A 

e 

10 

f 

18 

ef 

18 

a 

38 

ae 

46 

af 

36 

aef 

34 

b 

33 

be 

40 

bf 

31 

bef 

32 

ab 

56 

abe 

74 

abf 

74 

abef 

69 

c 

15 

ce 

13 

cf 

11 

cef 

10 

ac 

42 

ace 

34 

acf 

41 

acef 

43 

be 

34 

bee 

30 

bef 

46 

beef 

39 

abc 

69 

abce 

72 

abef 

47 

abcef 

44 

d 

57 

de 

53 

df 

45 

def 

37 

ad 

66 

ade 

75 

adf 

66 

adef 

60 

bd 

79 

bde 

70 

bdf 

56 

bdef 

67 

abd 

67 

abde 

83 

abdf 

88 

abdef 

86 

cd 

61 

ede 

49 

cdf 

47 

edef 

48 

acd 

78 

aede 

72 

aedf 

67 

aedef 

75 

bed 

67 

bede 

56 

bedf 

65 

bedef 

70 

abed 

95 

abede 

89 

abedf 

82 

abedef 

89 

Ana  1 vs  l.s 

of  Vari.iiu'O  T.ihlo 

for  t lio  K.nt  i ro 

' D.ita  Sol 

AN  OVA 

Soiifoi' 

S8 

OK 

MS 

KOM 

8(0)1.  1h 

1 

8641. 16 

K.iti’  ol  Kiro 

6201 . 16 

1 

6201 . 16 

1 . 00 

1 

1 . 00 

Si'c  t or 

14460.  0(i 

1 

14460.  0(i 

Thro.it  Array* 

286.371 

3 

41.46 

Kr  ror 

2647.441 

16 

48.  16 

Tot  a 1 

23342. 

6) 

* 

SS 

Throat  Array 

SS|,  + SSj,  + SS|.j, 

■k-k 

Sinnit'ioant  .it 

1%  lovol  . 

M 


(' 

180.  ')** 


128.7** 

. 02 

100. 2** 
1 . ^)8 
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APPENDIX  B 


SCREENING  ANALYSIS 


This  appendix  contains  an  analysis  of  the  results  after  each 
block  of  eight  experiments  was  run  and  a comparison  of  the  effects  to 
determine  relative  significance. 


A.  Block  1: 


B.  Block  2: 


2 = A + BD  + CE  = 62/3  = 20.67  £'  = -A  + BD  + CE  = -4A.67 


£„  = B + AD  + CE  = 31.33 

D 

2^  = C + AE  + BE  = 10.00 
2q  = D + AB  + EF  = 39.33 


2^  = -B  + AD  + CF  = -24.67 
2'  = -C  + AE  + BF  = 7.33 

Li 

2^  = -D  + AB  + EF  = -27.33 


2^  = E + AC  + DF  = -10.00 
E 


2'  = -E  + AC  + DF  = 2.00 
E 


2^  = F + BC  + DE  = -4.67 
F 


2'  = -F  + BC  + DE  = 7.33 
F 


2^,p=  CD  + BE  + AF  = 4.00 


C.  Block  1: 

^.(2.  + 2; 


2^D  = CD  + be  + AF  = 18.67 


C.  Block  2 


''a(2,  - 2 ) 


BD  + CE  = 12.00 


A = 32.67 


AD  + CF  = 3.33 


B = 28.00 


AE  + BF  = 8.67 


C = 1.33 


AB  + EF  = 6.00 


D = 33.33 


AC  + DF  = -4.00 


E = -6.00 


BC  + DE  = 1.33 


F = -6.00 
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1 

D . Block  3: 

! a*  = A - BD  + CE  = 4A.00 

! ^ 

; = -B  + AD  + CF  = -39.33 

D 

e*  = C + AE  - BF  = 2.67 

5*  = D - AB  + EF  = 36.00 

^*  = E + AC  + DF  = 2.00 

Z,*  = F - BC  + DE  = 4.00 

F 

I*  = CD  = BE  + AF  = 3.33 


D.  Blocks  1 and  3: 


h(i^.  + !>-*) 

S(!i.  - 

A + CE  = 32.34 

BD  = -11.67 

AD  + CF  = -4.00 

B = 35.33 

C + AE  = 6.34 

BF  ^ 3.67 

D + EF  = 37.67 

AB  = 1.67 

F + DE  = -.34 

BC  = -4.34 

CD  + AF  = 3.67 

BE  = .34 

E.  Block  4: 

V=  A + BD  - CE  = 34.00 
A 

= B + AD  - CF  = 16.67 

9.'^  = -C  + AE  + BF  = 2.00 

= D + AB  + EF  = 46.00 

II  = E - AC  + DF  = -4.67 
E 

= F - BC  + DE  = -7.33 

= -CD  + BE  + AF  = -2.00 
CD 


F.  Block  1: 

^2(£.  + 

A + BD  = 27. 3A 
B + AD  = 24.00 
AE  + BF  = 6.00 
D + AB  + EF  = 42.67 
E + DF  = -7.34 
F + DE  = -6.00 
AF  + BC  = 1.00 


F.  Block  4: 


CE  = -6.67 
CF  = 7.33 
C = 4.00 


H.  0.  TERMS  = 
AC  = -2.67 
BC  = 1.33 
CD  = 3.00 


-3.33 


i 


i 

j 


ElBHHiMHIi 


■UKBfei 
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APPENDIX  C 

SEQUENTIAL  FACTORIAL  ESTIMATION 

This  appendix  applies  the  sequential  factorial  estimation  procedure 
discussed  in  Chapter  V to  the  data  from  the  example  used  in  Chapter  IV  in 
order  to  demonstrate  its  use. 

A.  After  the  first  block  of  eight  observations  is  made,  estimates  for 
the  coefficients  in  the  model: 

y . b„  + bjA  + bjB  A bjC  -b  b^D  + bjE  + b^F  + b^^BD  (C. 1) 

can  be  obtained  from  the  equation 

6 = (x'x)"^x’y  (C.2) 

For  the  first  block  of  eight  observations. 


1 

-1 

-1 

-1 

1 

1 

1 

-1 

— — 

37 

1 

1 

-1 

-1 

-1 

-1 

1 

1 

36 

1 

-1 

1 

-1 

-1 

1 

-1 

-1 

9 

y = 

40 

1 

1 

1 

-1 

1 

-1 

-1 

1 

87 

1 

-1 

-1 

1 

1 

-1 

-1 

-1 

61 

1 

1 

-1 

1 

-1 

1 

-1 

1 

34 

1 

-1 

I 

1 

-1 

-1 

1 

-1 

46 

1 

1 

1 

1 

1 

1 

1 

1 

89 

Resulting  in 


I 


53.75 


11.75 


14.75 


-3.75 


-1.75 


so  the  fitted  model  is: 


y = 53.75  + 7.75A  + 11.75B  + 3.75C  + 14.75D  - 3. 75E 


- 1.75F  + 7.75B 


(C.5) 


Since  the  variance-covariance  matrix,  (x'x)  ^ is 


(x'x)' 


(C.6) 


all  of  the  coeffic’ents  in  (C.5)  have  variance  equal  to  a /8  and  co- 


variances zero.  The  column  vector,  B,  given  by  C.4  now  becomes  B as 


In  equation  (5.3)  and  is  used  throughout  the  entire  next  block. 
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B.  The  ninth  observation  is  taken  at  treatment  combination  abc  with  an 

observed  response  of  69  so  the  required  data  for  equation  (5.3)  is 

N = 8,  n = 1,  q = 8,  m = 1,  tg  = [ 1 , 1 , 1 , 1 ,-l ,-l ,-l , -1 ] , = 69, 

T 

y = r B = 60,  and  B as  given  in  (C.4). 

9 9 

This  results  in 


57.5 

11.5 

15.5 

7.5 

18.5 
0 

2.0 

11.5 


(C.7) 


so  the  fitted  model  after  nine  observations  is 


y = 57.5  + 11. 5A  + 15. 5B  + 7.5C  + 18. 5D  + 2. OF  + 11.5  BD 


(C.8) 


and  the  variance  of  the  coefficients  in  (C.8)  as  obtained  from  (5.4) 
2 

is  15/128  O but  the  covariances  are  no  longer  zero  because  the 

columns  of  the  design  matrix  are  no  longer  orthogonal.  The  co- 

variances  can  be  obtained  from  the  approximate  entry  in  the 

variance-covariance  matrix,  (x'x)  This  procedure  would  be  con- 

th  * 

tinned  through  the  16  observation.  The  B obtained  after  applying 
(5.3)  to  the  16th  observation  would  become  the  values  of  B used 
throughout  the  third  block  of  eight  observations. 
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